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Abstract 


To  improve  operational  effectiveness  for  the  Canadian  Forces  (CF),  the  Joint  Unmanned  Aerial 
Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project  is  acquiring  a  medium 
altitude,  long-endurance  (MALE)  uninhabited  aerial  vehicle  (UAV).  In  support  of  the  JUSTAS 
project,  Defence  Research  and  Development  Canada  (DRDC)  -  Toronto  carried  out  a  literature 
review  on  relevant  auditory  and  tactile  displays  in  UAV  simulation  environments  for  the  ground 
control  station  (GCS)  interface  in  an  effort  to  identify  techniques  to  overcome  human  factors 
issues  in  the  supervisory  control  of  UAVs.  This  literature  review  was  used  to  generate  a  number 
of  different  design  proposals  for  multimodal  displays  (i.e.,  the  presentation  of  information  in  the 
auditory,  tactile,  and  visual  modalities).  Two  of  these  displays,  an  engine  RPM  sonification  and 
aggregate  attitude  upset  tactile  display,  were  chosen  for  further  development  and  testing.  These 
two  interfaces  were  implemented  in  a  GCS  simulator.  A  proposal  for  testing  the  displays  was 
provided  as  a  starting  point  for  further  discussion. 
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Executive  summary 


Baseline  and  Multimodal  UAV  GCS  Interface  Design:  Progress 
Report 

[Wayne  Giang,  Ehsan  Masnavi,  Sharaf  Rizvi,  Plinio  Morita,  Catherine  Burns]; 
DRDC  Toronto  CR  [enter  number  only:  9999-999];  Defence  R&D  Canada  - 

Toronto;  March  2011. 

Background:  An  uninhabited  aerial  vehicle  (UAV)  is  an  aircraft  system  without  an  onboard  pilot 
or  crew.  The  UAV  is  controlled  from  a  ground  control  station  (GCS).  Today's  UAVs  are  highly 
automated  and  to  some  extent,  autonomous.  Some  UAVs  can  perform  automated  take-off  and 
landing  (e.g.,  the  CU-170  Heron  used  by  the  Canadian  Forces).  UAV  developers  argue  that 
automation  and  autonomy  provide  several  benefits:  (a)  increased  flight  safety;  (b)  simplified 
operations;  (c)  lower  operating  costs;  and  (d)  reduced  operator  workload  (Attar,  2005).  However, 
these  benefits  are  not  always  realized.  Along  with  the  benefits  of  automation,  some 
disadvantages  occur  such  as  loss  of  situation  awareness  (Endsley  and  Kiris,  1995;  Endsley,  1996), 
loss  of  supervisory  control  (Parasuraman,  Molloy,  Mouloua,  &  Hilbum,  1996;  Sheridan,  1987), 
information  deprivation  that  occurs  from  remote  operations  (Manning,  Rash,  LeDuc,  Noback,  & 
McKeon,  2004),  and  high  workload  levels  for  operators  (Lee,  2008;  Woods,  1996).  These  issues 
point  to  the  need  for  improved  interfaces  to  help  operators  remain  in  the  loop  and  maintain 
situation  awareness  during  the  remote  monitoring  tasks  typical  of  UAV  monitoring. 

Results:  A  literature  review  was  completed  on  relevant  auditory  and  tactile  displays  in  UAV 
simulation  environments.  This  literature  was  used  to  generate  a  number  of  different  design 
proposals  for  multimodal  displays.  Two  of  these  displays,  an  engine  RPM  sonification  and 
aggregate  attitude  upset  tactile  display,  were  chosen  for  further  development  and  testing.  These 
two  interfaces  were  implemented  using  the  hardware  provided  by  DRDC-Toronto.  A  proposal  for 
testing  the  displays  was  provided  as  a  starting  point  for  further  discussion. 

Significance:  The  work  described  in  this  report  is  in  support  of  the  Canadian  Forces  (CF)  Joint 
Unmanned  Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  The 
JUSTAS  project  entails  the  acquisition  of  a  medium-altitude,  long-endurance  (MALE)  UAV. 
This  work  investigates  the  efficacy  of  a  multimodal  GCS  display  for  enhancing  supervisory 
control  for  automated  landing.  It  is  expected  that  the  findings  from  this  study  will  have 
implications  for  the  design  requirements  of  a  GCS  interface  for  the  JUSTAS  project. 

Future  plans:  Overall,  these  activities  have  allowed  us  to  reach  a  state  of  readiness  to  move  on  to 
the  next  stage  of  the  project.  The  next  phase  of  the  project  involves  refining  the  design  of  the 
engine  RPM  sonification  and  aggregate  attitude  upset  displays,  conducting  the  evaluation  of  the 
displays,  and  conducting  the  baseline  GCS  pilot  study. 
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1  Introduction 


An  uninhabited  aerial  vehicle  (UAV)  is  an  aircraft  system  without  an  onboard  pilot  or  crew.  The 
UAV  is  controlled  from  a  ground  control  station  (GCS).  Today's  UAVs  are  highly  automated  and 
to  some  extent,  autonomous.  UAVs  can  be  directed  to  follow  a  pre-programmed  mission;  they 
can  fly  to  designated  waypoints,  fly  specific  patterns,  correct  for  course  deviations  and  hold 
above  a  particular  coordinate  or  target.  Some  UAVs  can  perform  automated  take-off  and  landing 
(e.g.,  the  CU-170  Heron  used  by  the  Canadian  Forces).  UAV  developers  argue  that  automation 
and  autonomy  provide  several  benefits:  (a)  increased  flight  safety;  (b)  simplified  operations;  (c) 
lower  operating  costs;  and  (d)  reduced  operator  workload  (Attar,  2005).  However,  these  benefits 
are  not  always  realized.  Along  with  the  benefits  of  automation,  some  disadvantages  occur  such  as 
loss  of  situation  awareness  (Endsley  and  Kiris,  1995;  Endsley,  1996),  operators  being  forced  to 
perform  supervisory  control  tasks  (Parasuraman,  Molloy,  Mouloua,  &  Hilbum,  1996;  Sheridan, 
1987),  information  deprivation  that  occurs  from  remote  operations  (Manning,  Rash,  LeDuc, 
Noback,  &  McKeon,  2004),  and  high  workload  levels  for  operators  (Lee,  2008;  Woods,  1996). 
These  issues  point  to  the  need  for  improved  interfaces  to  help  operators  remain  in  the  loop  and 
maintain  situation  awareness  during  the  remote  monitoring  tasks  typical  of  UAV  monitoring. 

The  work  described  in  this  report  is  in  support  of  the  Canadian  Forces  (CF)  Joint  Unmanned 
Aerial  Vehicle  Surveillance  Target  Acquisition  System  (JUSTAS)  project.  The  JUSTAS  project 
entails  the  acquisition  of  a  medium-altitude,  long-endurance  (MALE)  UAV.  This  work 
investigates  the  efficacy  of  a  multimodal  GCS  display  for  enhancing  supervisory  control  for 
automated  landing.  It  is  expected  that  the  findings  from  this  study  will  have  implications  for  the 
design  requirements  of  a  GCS  interface  for  the  JUSTAS  project. 

This  report  is  a  mid-project  progress  report  detailing  the  work  that  has  occurred  between 
September  2010  and  March  2011  in  order  to  prepare  for  a  large  scale  study  of  two  GCS  designs. 
One  of  the  GCS  designs  will  be  a  visual  interface  (called  the  “baseline  condition”),  essentially 
simulating  a  typical  UAV  visually  based  GCS  interface.  The  other  design  will  be  a  multimodal 
interface  where  tactile  and  auditory  information  will  be  added  to  the  interface  to  see  if  this  new 
information  can  improve  operator  performance  and  situation  awareness. 

The  objective  of  this  report  is  to  outline  potential  designs  for  the  auditory  and  tactile  information 
of  the  eventual  multimodal  interface.  To  meet  this  objective,  the  current  auditory  and  tactile 
interfaces  for  UAV  control  are  described  in  Section  2.  Drawing  from  this  literature,  new  auditory 
and  tactile  designs  are  proposed  in  Section  3. 


2  Auditory  and  Tactile  Displays  for  Enhancing 
Monitoring  Automated  UAVs 


In  the  control  of  automated  UAVs,  GCS  operators  are  responsible  for  monitoring  the  status  of  the 
UAV  and  ensuring  that  safe  operation  is  possible.  This  task  is  important  during  the  events  leading 
up  to  the  recovery  of  the  UAV..  When  abnormal  events  occur  that  may  jeopardize  the  safe 
recovery  of  the  UAV,  operators  are  able  to  abort  the  landing.  These  decisions  require  information 
about  the  current  status  of  the  vehicle  (e.g.,  the  current  altitude  and  attitude),  the  environmental 
conditions,  and  the  current  tactical  situation.  Therefore,  it  is  important  that  UAV  operators  have 
access  to  and  are  able  to  process  this  information  even  under  heavy  workload  situations  in  which 
they  may  also  be  responsible  for  other  critical  mission  tasks  such  as  examining  sensory  imagery 
and  monitoring  communications  channels. 

Multimodal  interfaces  are  one  possible  solution  to  this  challenging  task  of  supporting  the 
information  needs  of  GCS  operators.  The  visual  channel  is  often  overloaded  by  information  in 
current  UAV  GCSs.  The  auditory  and  tactile  sensory  modalities  are  plausible  alternatives  for 
information  presentation  and  attention  cueing.  In  the  following  sections  we  will  examine  the 
current  literature  as  it  has  discussed  the  use  of  auditory  and  tactile  displays  in  UAV  simulation 
environments.  We  will  then  provide  a  review  of  the  auditory  and  tactile  displays  proposed  for 
improving  operator  awareness  during  UAV  autoland  situations. 

2.1  Literature  on  Auditory  and  Tactile  Displays  in  UAV 
Simulation  Environments 

To  date,  there  are  few  studies  that  have  examined  auditory  and  tactile  displays  in  UAV  simulation 
environments.  Studies  involving  auto-landing  systems  and  automated  flight  are  even  fewer  still. 
In  this  section  we  discuss  some  of  these  studies  and  examine  similarities  and  differences  between 
them.  We  define  UAV  simulation  environments  as  environments  where  the  task  is  to  control  or 
monitor  a  single  or  multiple  UAVs  through  a  simulated  mission.  We  have  identified  two  types  of 
simulations  in  the  literature:  those  that  involved  manual  control  using  a  joystick  or  other  analog 
methods,  and  those  that  relied  on  point-and-click  interfaces  and  had  a  higher  focus  on  supervisory 
control. 

2.1.1  Manual  Control  UAV  Simulations 

In  manual  control  situations,  participants  are  often  required  to  make  small  course  changes  in 
response  to  deviations  from  flight  paths  or  to  correct  glide  slopes.  We  identified  three  papers  that 
made  use  of  this  type  of  simulation.  Calhoun  et  al.  (2003;  2004)  used  a  UAV  simulation 
environment  consisting  of  a  display  with  map  and  mission  information,  simulated  video  imagery 
from  the  UAV,  and  a  third  display  for  communications.  Participants  were  asked  to  maintain  the 
correct  heading  and  airspeed  of  the  UAV  using  a  control  stick  and  throttle  control.  The  manual 
control  of  the  UAV  was  largely  dependent  on  the  visual  information  presented  by  the  interface; 
however,  warnings  were  presented  through  visual,  auditory,  and  tactile  stimuli.  In  Calhoun  et  al. 
(2003),  non-spatial  alert  cues  were  presented  through  the  use  of  two  tactors,  one  located  on  each 
of  the  participant’s  wrists.  There  were  three  levels  of  warnings  that  participants  were  required  to 
respond  to:  non-critical  warnings,  critical  warnings,  and  information  queries.  Figure  2.1  below 


shows  the  different  types  of  alerts  used  for  each  of  these  warnings  in  the  baseline  (visual  and 
auditory)  and  tactile  (tactile,  visual,  and  auditory)  alert  conditions. 


TASK 

DIFFICULTY 

ALERT  CONDITION 

LOW 

HIGH 

BASELINE 

TACTILE 

UAV  Turns 

1 

3 

Normal 

Operations 

2 

4 

No  alerts  for  routine  in-flight  and  waypoint  update  tasks 

Non-Critical 

Warnings 

1 

3 

Visual:  Colored  “A”  or 
“D”  on  HUD 

Auditory:  Yes* 

Visual:  Colored  “A”  or  “D”  on  HUD 
Auditory:  Yes* 

Tactile:  None;  tactile  reserved  for 
Critical  Warnings  &  Information 
Queries 

Critical 

Warnings 

3 

3 

Visual:  Colored  “A”  or 
“D”  on  HUD 

Auditory:  Yes* 

Visual:  Colored  “A"  or  “D”  on  HUD 
Auditory:  Yes* 

Tactile:  One  tactor  vibrated: 
left-arm  tactor:  icing 
right-arm  tactor:  servo  overheat 

Information 

Queries 

2 

2 

Visual:  Red  “QUERY” 
on  HUD 

Auditory:  None 

Visual:  None 

Auditory:  None 

Tactile:  Both  tactors  vibrated 

*  Auditory  Alert:  0.4  second  klaxon  sound,  primarily  at  487  Hz;  10.3  dB. 


Figure  2.1:  Alert  conditions  used  in  Calhoun  et  al.  (2003,  p.  2120). 

Auditory  information  was  presented  in  all  alert  conditions.  The  different  levels  of  information 
were  coded  using  the  two  tactors  by  varying  the  number  of  tactors  vibrating  and  the  location  of 
the  vibrating  tactor.  This  limited  the  tactile  signals  to  only  three  levels  (left  arm  tactor  for  icing, 
right  arm  tactor  for  servo  overheat,  and  both  tactors  for  information  queries).  The  tactors  were 
mounted  on  the  insides  of  the  wrists  and  were  placed  1.5  inches  from  the  bottom  of  the 
participant’s  palms.  The  tactors  were  driven  using  a  5  volt  square  wave  with  a  50%  duty  cycle 
that  had  a  frequency  that  changed  from  250  to  500  Hz  over  a  0. 1  second  interval. 

Calhoun  et  al.  (2003)  found  that  the  tactile  condition  produced  significantly  faster  responses  than 
the  baseline  condition  for  information  queries,  but  not  for  any  of  the  other  tactile  alerts.  Calhoun 
et  al.  (2003)  suggested  that  this  may  have  been  because  two  tactors  were  used  for  the  information 
query  alert,  while  only  one  tactor  was  used  for  the  warnings.  However,  the  two  warning 
conditions  were  also  accompanied  by  an  auditory  alert  while  the  information  query  condition 
used  only  a  tactile  alert  (in  addition  to  the  visual  alerts  provided  in  all  conditions).  The  authors 
caution  that  this  study  could  not  provide  further  insight  into  whether  the  significantly  faster 
responses  were  created  by  having  two  vibrating  tactors  instead  of  just  a  single  vibrating  tactor  or 
the  absence  of  the  redundant  auditory  cue.  The  information  query  tactile  alert  was  entirely  tactile 
while  the  baseline  condition  was  entirely  auditory.  The  warning  alerts  featured  multi-sensory 
alerts  that  used  vision,  audition,  and  touch  in  the  tactile  condition.  While  it  was  possible  that  the 
use  of  tactile  alerts  could  prove  to  be  distracting,  especially  when  two  (auditory  and  visual) 
sensory  modalities  were  already  employed,  the  results  suggested  that  tactile  cues,  in  the  presence 
of  auditory  and  visual  cues  did  not  hinder  or  improve  performance.  However,  uni-modal  tactile 
cues  (the  tactile  information  queries  condition)  did  produce  faster  response  times  than  uni-modal 
visual  cues  (the  baseline  information  queries  condition).  The  participants’  performance  during  the 
tracking/flight  task  was  also  analyzed,  however  no  significant  effects  of  alert  types  were  found. 
This  led  the  authors  to  conclude  that  the  introduction  of  the  tactile  cue  did  not  allow  the 
participants  to  direct  more  resources  to  the  flight  task. 

In  a  subsequent  study,  Calhoun  et  al.  (2004)  used  a  similar  UAV  simulation  task  but  changed  the 
type  of  auditory  and  tactile  alerts  used.  The  tactors  were  again  placed  on  the  inside  of  both  wrists. 
However,  instead  of  using  three  different  alerts  (left  wrist,  right  wrist,  and  both  tactors  at  the  same 
time),  the  authors  used  only  one  level  of  tactile  stimulation  which  activated  both  tactors.  This 


change  from  the  previous  study  (Calhoun  et  al.,  2003)  was  based  on  the  significant  difference  in 
response  time  produced  in  solely  the  two-tactor  activation  condition.  The  auditory  stimuli 
consisted  of  two  different  levels.  Type  1  alerts  were  klaxon-sounds  of  0.4  seconds,  primarily  at 
487Hz,  presented  at  10.3dB.  Type  2  alerts  were  klaxon-sounds  with  duration  of  1.1  seconds, 
primarily  at  500Hz,  presented  at  9.0dB.  The  first  alert  type,  Type  1,  was  used  across  all  alert 
conditions  (baseline,  +aural,  and  +tactile).  The  tactile  alerts  and  the  Type  2  auditory  alerts  were 
reserved  for  the  critical  alert  conditions,  while  the  caution  alert  conditions  only  made  use  of  a 
visual  alert  and  the  Type  1  auditory  alert.  A  list  of  these  conditions  is  shown  in  Figure  2.2.  The 
authors  concluded  that  redundant  non-visual  alerts  improved  performance  over  using  just  the 
visual  alerts.  They  also  found  that  aural  alerts  were  just  as  effective  as  tactile  alerts,  even  when 
varying  conditions  of  auditory  load. 
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Figure  2.2:  Alert  conditions  for  Calhoun  et  al.  (2004,  p.  138). 

In  another  experiment,  Aretz  et  al.  (2006)  had  participants  manually  land  a  UAV.  Participants 
used  controls  and  visual  displays  that  were  similar  to  those  found  in  the  studies  by  Calhoun  et  al. 
(2003,  2004).  However,  participants  were  also  presented  vibrotactile  feedback  for  altitude 
deviations  through  a  tactor  vest  with  four  rows  of  tactors.  Each  of  the  rows  represented  a  different 
level  of  deviation  from  optimal  altitude  during  the  approach.  The  top  most  row  would  vibrate 
intensely  (200ms  on,  1 00ms  off,  66%  duty  cycle)  if  the  UAV  was  20  feet  above  the  optimal  glide 
slope.  The  second  highest  row  would  vibrate  softly  (100ms  on,  600ms  off,  -14%  duty  cycle) 
when  the  UAV  was  10  feet  above  the  optimal  glide  slope.  If  the  glide  slope  deviation  was  below 
10  feet,  then  the  vest  would  not  vibrate  at  all.  Therefore,  the  magnitude  of  the  glide  slope 
deviation  was  coded  by  both  the  location  of  the  vibrating  tactors  and  the  duty  cycle.  Higher 
deviation  was  mapped  onto  a  longer  duty  cycle.  A  similar  coding  strategy  was  used  for  the 
bottom  two  rows  for  when  the  UAV  was  below  the  optimal  glide  slope,  with  lower  locations  and 
higher  duty  cycles  representing  larger  glide  slope  deviation.  The  type  of  signal  used  to  drive  the 
tactors  was  not  mentioned  in  the  article.  Aretz  et  al.  (2006)  found  that  tactile  feedback  changed 
how  quickly  participants  were  able  to  achieve  good  performance  in  the  landing  task.  Even  when 
operators  were  trained  without  tactile  feedback,  a  tactile  feedback  system  that  was  designed  to  be 
easily  understandable  could  increase  performance.  It  is  important  to  note  that  this  was  an  applied 
study  that  did  not  focus  on  how  the  tactile  signals  would  be  perceived  at  a  basic  level.  The 
implications  of  basic  tactile  perception  research  on  this  tactile  design  will  be  discussed  in  Section 
2.1.4. 


2.1.2  Monitoring  and  Supervisory  Control  UAV  Scenarios 


In  supervisory  control  scenarios,  the  participant’s  main  task  was  not  the  manual  control  of  the 
UAV  through  analog  inputs.  Instead,  the  UAVs  in  these  simulations  were  largely  autonomous  and 
much  of  the  flight  was  not  controlled  by  the  participants,  whose  main  goal  was  to  monitor  the 
UAVs  for  abnormal  behaviour  and  to  make  decisions  about  higher  level  mission  goals.  A  pilot 
study  by  Donmez,  Graham,  and  Cummings  (2008)  examined  the  use  of  haptic  peripheral  displays 
for  the  control  of  multiple  UAVs.  The  Multiple  Autonomous  Unmanned  Vehicle  Experimental 
(MAUVe)  test  bed  was  used  for  this  experiment.  This  experimental  platform  used  a  multi¬ 
monitor  visual  display,  an  over-the-head  headset  for  auditory  information,  and  an  inflatable 
pressure  vest  and  vibrating  wristbands  for  haptic  and  tactile  information.  The  goal  of  the 
experiment  was  to  examine  the  differences  between  continuous  feedback  and  threshold  feedback 
haptic  and  tactile  displays. 

Two  variables  were  chosen  to  be  represented  using  the  vest  and  wrist  displays:  late  arrivals  and 
course  deviations.  Late  arrivals  were  displayed  haptically  through  the  inflatable  vest.  If  a  UAV 
was  projected  to  be  arriving  late  at  a  way-point,  the  vest  was  inflated  to  varying  degrees  based  on 
the  priority  of  the  target.  Only  a  few  of  the  air  bladders  on  the  vest  were  inflated  for  low  priority 
targets  (4  bladders).  Progressively  more  air  bladders  were  inflated  for  the  medium  (6  bladders) 
and  high  priority  targets  (8  bladders).  Each  of  the  air  bladders  inflated  to  20psi.  In  the  continuous 
feedback  condition,  the  vest  would  stay  inflated  until  the  participant  responded  to  the  late  arrival 
or  the  UAV  reached  its  target.  However,  in  the  threshold  feedback  condition,  the  entire  vest 
would  stay  inflated  for  a  2000ms  interval  after  the  detection  of  the  late  arrival.  The  tactor 
wristband  worn  on  the  participant’s  right  hand  was  used  to  represent  information  about  the  course 
deviations  of  UAVs.  The  five  wrist  tactors  in  the  wristband  were  activated  when  course 
deviations  occurred  and  had  a  frequency  of  100Hz.  No  information  was  given  regarding  the 
amplitude  or  the  type  of  waveform  (e.g.,  sine,  square,  or  saw-tooth)  used  to  drive  the  tactors.  In 
the  continuous  feedback  condition,  the  tactors  would  stay  on  for  300ms  per  activation.  As  the 
deviation  increased,  the  number  of  tactors  activated  was  increased  and  the  gap  time  between 
activations  decreased.  This  led  to  a  gradual  increase  in  duty  cycle  for  large  course  deviations.  A 
graph  with  the  number  of  tactors  activated  and  gap  times  used  can  be  seen  in  Figure  2.3.  In  the 
threshold  feedback  for  course  deviations,  all  the  tactors  would  be  activated  for  600ms  whenever 
the  course  deviation  surpassed  10  degrees. 


Figure  2.1:  Number  of  motors  activated  and  gap  time  between  each  activation  as  a  function  of 

UAV  course  deviation  (Donmez,  2008,  p.  9). 


Continuous  haptic  feedback  was  found  to  produce  significantly  faster  response  times  for  course 
deviations,  while  threshold  haptic  feedback  produced  faster  response  times  for  late  arrivals.  No 
significant  effects  of  feedback  type  were  found  for  the  secondary  auditory  loading  task  or  for 
subjective  measures  of  workload  (as  measured  by  the  NASA  Task  Load  Index;  NASA-TLX).  The 
authors  suggest  that  continuous  information,  such  as  a  deviation  from  the  course,  is  best 
supported  using  continuous  feedback,  while  a  discrete  event,  such  as  a  late  arrival,  is  best 
supported  using  threshold  feedback.  The  post-test  feedback,  gathered  using  the  NASA-TLX, 
revealed  that  participants  preferred  the  threshold  feedback  for  course  deviations  over  continuous 
feedback,  even  though  threshold  feedback  produced  slower  reaction  times.  The  authors  state  that 
this  mismatch  between  the  subjective  and  actual  performance  could  be  due  to  participants’ 
annoyance  at  the  continual  buzzing  of  the  wristband. 

A  similar  study  from  the  same  lab  was  conducted  by  Graham  and  Cummings  (2007)  but  used 
auditory  peripheral  displays  instead  of  haptic  displays.  The  same  simulation,  MAUVe,  was  used 
for  this  study,  and  the  goal  was  again  to  examine  the  differences  in  continuous  and  discrete 
representation  of  variables.  Late  arrivals  and  course  deviations  were  the  variables  that  were 
chosen  to  be  sonified,  and  for  each  of  the  variables  a  continuous  and  discrete  sonification  was 
developed.  The  continuous  late  arrival  sonification  was  composed  of  discrete  harmonic  signals 
that  were  continuous  played  once  the  system  predicted  the  late  arrival  of  a  UAV.  The  sonification 
continued  until  the  operator  responded  to  the  late  arrival  event  or  until  the  UAV  reached  the  way 
point.  Graham  and  Cummings  described  the  signals  as: 

“composed  of five  Formant  filters  that  were  applied  to  a  mix  of  pink  noise  and 
the  ambient  signal.  During  the  condition  of  no  late  arrivals,  a  baseline  audio 
condition  was  generated  with  two  filters  set  to  261.6Hz,  two  filters  set  at 
329.4Hz,  and  one  filter  set  at  392.0Hz  (a  major  C  triad  with  extra  balance  in 
the  root  and  third).  If  it  was  late  to  a  low  priority  target,  a  signal  was 
generated  with  two  filters  set  to  261.6Hz,  two  filters  set  at  311.1Hz,  and  one 
filter  set  at  392.0Hz  (a  minor  C  triad  with  extra  balance  in  the  root  and  third). 

If  it  was  late  to  a  medium  priority  target,  a  signal  was  generated  with  three 
filters  set  to  293.6Hz  and  two  filters  set  at  415.3Hz  (a  tritone  interval  with  a  D 
root).  If  the  UAV  was  predicted  to  be  late  to  a  high  priority  target,  a  signal 
was  generated  with  three  filters  set  to  369.9Hz  and  two  filters  set  at  523.25Hz 
(a  tritone  interval  with  an  F#  root).  As  the  priority  increased,  the  pink  noise 
mix  also  increased,  from  0.25  for  the  baseline,  to  0.7,  0.8,  and  1.0  for  the 
three  priority  levels  ”  (Graham  and  Cummings,  2007,  p.  8). 

The  discrete  late  arrival  sonification  used  a  signal  tone  with  a  fundamental  frequency  of  415Hz 
which  played  for  18ms.  This  signal  was  used  whenever  the  system  predicted  the  late  arrival  of 
one  of  the  UAVs. 

The  continuous  course  deviation  sonification  was  created  using: 

“comb  filters  that  were  applied  to  a  mix  of  pink  noise  and  the  ambient 
signal....  The  mix  ranged  from  0.2  pink  noise  for  low  deviation  to  0.9  pink 
noise  for  high  deviation.  The  comb  filters  had  a  base  delay  of  0.2ms,  with  a  50 
percent  each  mix  of  the  base  signal  and  a  feed  forward  delay.  The  delay 
values  were  then  oscillated  to  create  a  periodic  signal.  Because  this  was  a 
continuous  audio  scheme,  it  played  continually  to  provide  an  auditory >  image 
of  UAV  path  position.  As  a  UAV  drifted  further  off  course,  the  frequency  of 


oscillation  of  the  comb  filter  delay  decreased  from  17 Hz  to  4.5Hz,  and  the 
depth  of  oscillation  increased  from  0.2ms  to  0.7ms,  thus  changing  on  a 
continual  scale”  (Graham  &  Cummings,  2007,  p.  8). 

The  discrete  course  deviation  alert  made  use  of  a  single  tone  with  a  fundamental  frequency  of 
1000Hz  which  played  for  8ms.  This  signal  was  used  whenever  a  UAV  deviated  from  the  planed 
course.  The  results  were  very  similar  to  the  haptic  study  (Donmez  et  al.,  2008);  continuous 
representations  were  best  suited  for  continuous  data  and  discrete  representations  were  best  suited 
for  discrete  data. 

2.1.3  Manual  Control  UAV  Simulations 

In  this  section  we  examine  the  auditory  and  tactile  cues  used  in  5  different  UAV  simulation 
environments.  The  multimodal  stimuli  were  used  in  a  wide  variety  of  tasks  and  situations.  This 
suggests  that  there  are  many  possibilities  for  the  use  of  auditory  and  tactile  cues  within  the 
domain  of  supporting  UAV  autolanding.  Auditory  and  tactile  cues  and  displays  were  used  in  both 
situations  where  an  operator  was  required  to  manually  fly  a  UAV  using  an  analog  control,  and 
when  the  operator  was  only  responsible  for  monitoring  the  overall  status  of  the  UAV.  The 
following  table  contains  the  different  types  of  codes  used  for  each  modality. 


Table  1:  Coding  used  in  UAV  simulation  studies. 


Modality 

Coding  Used 

Variables  Coded 

Number  of  Levels 

or  Range 

Studies 

Tactile 

Spatial  location, 
horizontal  rows 

on  a  vest. 

Glide  slope 
deviation  (horizontal 
deviation). 

5 

Aretz  et  al.  (2006) 

Duty  cycle  of 
vibrotactile 
stimuli. 

Glide  slope 
deviation  (horizontal 
deviation). 

5 

Aretz  et  al.  (2006) 

Course  deviation. 

10+ 

Donmez  et  al. 
(2008) 

Number  of 
tactors  activated. 

Course  deviation. 

5 

Donmez  et  al. 
(2008) 

Tactors 

activated. 

Course  deviation 
occurred. 

1 

Donmez  et  al. 
(2008) 

Haptic 

Number  of  air 
bladders 
inflated. 

Priority  level  of  a 
late-arriving  UAV. 

4 

Donmez  et  al. 
(2008) 

Haptic  vest 
inflated. 

UAV  late  arrival 
detected. 

1 

Donmez  et  al. 
(2008) 

Warnings. 

1 

Calhoun  et  al. 
(2003) 

Klaxon  sounds. 

Alert  conditions 
(warnings  and 
information 
queries). 

2 

Calhoun  et  al. 
(2004) 

Different 
musical  chords 
and  intervals. 

Priority  level  of  a 
late-arriving  UAV. 

4 

Graham  and 
Cummings  (2007) 

Priority  level  of  a 
late-arriving  UAV. 

4 

Graham  and 
Cummings  (2007) 

Auditory 

Increased  mix  of 
pink  noise. 

Course  deviation. 

Range  from  0.2- 
0.9  as  deviation 
increased  (number 
of  levels 
unknown). 

Graham  and 
Cummings  (2007) 

UAV  late  arrival 
detected. 

1 

Graham  and 
Cummings  (2007) 

An  auditory 
beep. 

Course  deviation 
detected. 

1 

Graham  and 
Cummings  (2007) 

Comb  filter 
delay  oscillation 
frequency. 

Course  deviation. 

Range  from  1 7Hz- 
4.5Hz  as  deviation 
increased  (number 
of  levels 
unknown). 

Graham  and 
Cummings  (2007) 

Comb  filter 
delay  oscillation 
depth. 

Course  deviation. 

Range  from 
0.2ms-0.7ms 
(number  of  levels 
unknown). 

Graham  and 
Cummings  (2007) 

Overall,  these  studies  demonstrated  that  certain  coding  methods  were  better  suited  for  certain 
applications.  Donmez  et  al.  (2008)  and  Graham  and  Cummings  (2007)  both  showed  that 
continuous  coding  methods  were  best  suited  for  continuous  data,  while  discrete  coding  methods 
were  best  suited  for  discrete  data.  While  this  type  of  coding  is  possible  with  some  auditory 
characteristics,  most  of  the  tactile  codes  were  not  continuous.  This  was  especially  true  for  tactile 
spatial  codes  which  relied  on  a  small  set  of  tactors.  However,  the  Donmez  et  al.  (2008)  study  also 
showed  that  participants  preferred  threshold  (discrete)  feedback  over  continuous  feedback.  This 
could  be  attributed  to  lower  perceived  workload,  because  participants  were  not  required  to 
discriminate  between  many  different  levels  in  threshold  conditions.  While  this  was  true  for  the 
monitoring  task  used  in  the  Donmez  et  al.  (2008)  study,  continuous  stimuli  may  be  better  suited 
for  supporting  tracking  tasks  where  participants  are  required  to  manually  control  and  compensate 
for  deviations. 

None  of  the  codes  used  caused  decreased  performance  when  compared  to  the  baseline  cases, 
which  were  often  based  solely  on  visual  feedback.  Duty  cycle  was  one  code  that  was  used  in 
multiple  studies  as  a  method  for  showing  increased  magnitude  of  a  monitored  variable.  Typically, 
higher  duty  cycles  were  mapped  onto  larger  magnitudes.  For  instance,  Aretz  (2006),  mapped 
higher  magnitude  glide  slope  deviations  to  a  66%  duty  cycle,  compared  to  only  a  14%  duty  cycle 
for  the  lower  magnitude  glide  slope  deviations.  Aretz  (2006)  also  used  spatial  codes  to  represent 
spatial  deviations,  which  could  be  an  intuitive  method  for  coding  spatial  information.  Participants 
found  that  tactile  feedback  changed  how  quickly  they  were  able  to  achieve  good  performance  in 
the  landing  task.  In  the  Graham  and  Cummings  (2007)  study,  larger  magnitudes  of  course 
deviation  and  higher  priority  targets  had  larger  amounts  of  pink  noise  mixed  into  the  auditory 
sonification.  This  suggests  that  “noisier”  sonifications  are  mapped  onto  data  of  higher  magnitudes 
and  intensities.  Because  higher  intensity  data  requires  a  more  immediate  response,  designers  may 
wish  to  code  urgent  information  with  increased  amounts  of  noise. 

In  conclusion,  while  there  are  still  very  few  studies  that  have  used  auditory  and  tactile  cueing  in 
UAV  simulation  environments,  we  can  draw  from  many  of  these  studies  to  build  a  foundation  for 
the  creation  of  our  own  multimodal  designs.  It  will  be  important  to  ensure  that  data  that  is 
continuous  will  be  mapped  onto  continuous  display  characteristics  and  that  discrete  display 
characteristics  are  used  to  represent  discrete  data.  Spatial  location,  duty  cycle,  the  amount  of  noise 
mixed  into  a  signal,  frequency,  and  magnitude  are  all  codes  that  can  be  used  in  the  design  of  our 
signals. 

2.1.4  Perceptual  Issues  of  Tactile  Perception 

In  the  previous  section,  we  reviewed  a  number  of  studies  that  have  made  use  of  auditory  and 
tactile  interfaces  to  present  information  during  UAV  simulations.  However,  one  drawback  of 
many  of  the  studies  discussed  was  that  they  did  not  make  use  of  current  research  on  the 
perceptual  issues  of  tactile  perception.  In  Giang  et  al.  (2010),  a  number  of  issues  related  to  the 
perception  of  tactile  stimuli  including  spatial  acuity,  adaptation,  and  habituation  were  discussed. 
In  this  section  we  review  the  current  literature  on  tactile  perception  with  regards  to  the  task  of 
presenting  course  deviation  information.  We  will  focus  on  the  merits  and  drawbacks  of  some  of 
the  coding  methods  discussed  in  Section  2.1. 

In  tactile  displays  there  are  a  few  major  issues  that  could  have  large  impacts  on  the  design  of  the 
displays.  In  this  section  we  will  discuss  three  of  these  factors:  spatial  acuity,  adaptation,  and 
habituation.  These  factors  were  chosen  because  they  impact  the  physical  layout  of  tactor  based 
tactile  displays.  Spatial  layout  and  tactile  pattern  design  are  two  design  parameters  that  are  very 
easy  to  manipulate  in  the  design  of  tactile  vests  and  have  much  more  perceptible  characteristics 
than  differences  in  frequency,  amplitude,  and  waveform.  Spatial  acuity  refers  to  an  individual’s 


ability  to  differentiate  between  two  different  tactile  stimuli  when  they  occur  at  different  spatial 
locations.  Spatial  acuity  varies  across  the  body  and  is  highest  for  the  fingers,  lips,  and  genitals 
(Cheung  et  al.,  2008).  Spatial  acuity  on  the  torso  is  much  lower,  and  for  vibrotactile  stimuli,  it  is 
relatively  uniform  over  the  entire  torso.  In  this  region  spatial  acuity  is  approximately  2-3cm  for 
vibrotactile  stimuli.  This  acuity  is  better  for  horizontally  oriented  arrays  located  in  line  with  the 
spine  and  navel  and  is  approximately  1  cm  in  these  regions  (Van  Erp  et  al.,  2005). 

Within  tactile  displays,  spatial  acuity  impacts  where  tactors  should  be  located.  In  a  tactile  vest, 
spatial  location  could  be  used  to  code  information  (e.g.,  Aretz  et  al.,  2006)  study.  When  this  type 
of  coding  is  used,  it  is  important  to  ensure  that  users  are  able  to  differentiate  between  the  different 
spatial  locations  by  ensuring  that  tactors  are  not  placed  too  closely  together.  Aretz  et  al.  (2006) 
did  not  indicate  the  inter-tactor  spacing  between  rows.  However,  as  mentioned  previously,  the 
spatial  acuity  on  the  torso  is  approximately  2-3  cm.  Therefore,  inter-tactor  spacing  should  be  at 
least  3  cm  for  proposed  tactile  vest  configurations.  Spatial  acuity  is  important  when  different 
tactor  locations  must  be  differentiated.  However,  groups  of  tactors  that  represent  a  single 
grouping  (such  as  the  rows  within  the  Aretz  et  al.  (2006)  study)  may  not  need  the  same  degree  of 
inter-tactor  spacing. 

Adaptation  is  another  important  factor  in  the  design  of  tactile  displays.  Adaptation  refers  to  “a 
change  in  the  percept  of  a  stimulus  after  prolonged  stimulation.”  (Cheung  et  al.,  2008,  p.  2-8)  In 
situations  when  the  tactor  vest  is  used  for  monitoring  a  continuous  variable,  care  must  be  taken  to 
reduce  the  effects  of  adaptation.  Adaptation  does  not  happen  across  frequency  bands  (Cheung  et 
al.,  2008),  so  one  method  for  preventing  adaptation  is  by  changing  the  tactor’s  frequency. 
However,  this  may  not  be  as  applicable  in  the  design  of  tactile  displays  because  frequency  is  not 
an  ideal  coding  parameter  as  suggested  by  our  previous  literature  review  (Giang  et  al.,  2010). 
Another  method  suggested  by  Cheung  et  al.  (2008)  was  the  use  of  appropriate  duty  cycles  to 
combat  adaptation.  A  duty  cycle  with  an  off  cycle  three  times  as  long  as  the  on  cycle  was  used  as 
an  example  (i.e.,  duty  cycle  of  25%).  In  the  Aretz  et  al.  (2006)  study  and  the  Donmez  et  al.  (2008) 
study,  course  deviation  was  coded  using  tactor  duty  cycles.  Specifically,  the  Aretz  et  al.  (2006) 
study  used  an  “intense”  duty  cycle  of  66%  and  a  “soft”  duty  cycle  of  16%.  The  Donmez  et  al. 
(2008)  study  used  a  variety  of  different  duty  cycles  that  gradually  increased  as  the  course 
deviation  increased.  For  both  these  studies,  only  high  degrees  of  course  deviation  would  result  in 
duty  cycles  that  could  possibly  suffer  from  adaptation.  Also,  since  both  of  these  occurred  in 
studies  that  had  manual  control  of  the  UAV,  the  course  deviation  would  likely  be  corrected  before 
adaption  would  occur.  Hollins  et  al.  (1990  as  cited  by  Cheung  et  al.,  2008)  state  that  the  time 
constant  of  adaption  is  approximately  2  minutes.  Thus,  if  high  course  deviations  are  corrected  in 
less  than  2  minutes,  adaptation  may  not  occur. 

Habituation  is  another  phenomenon  that  could  occur  when  tactile  patterns  are  presented  to  the 
user.  Unlike  adaptation,  habituation  does  not  degrade  sensory  sensitivity.  Instead,  a  sensation  that 
has  habituated  no  longer  draws  attention  to  itself  (Cheung  et  al.,  2008).  Cheung  et  al.  (2008)  state 
that  even  duty  cycles  that  can  avoid  adaptation  may  still  fall  victim  to  habituation.  Thus,  any 
continuous  signal  will  likely  be  habituated  to  over  a  long  duration  of  exposure.  The  authors 
suggest  that  “null  zone  conditions”,  conditions  that  indicate  nominal  behaviour,  should  be  created 
where  no  stimuli  is  presented.  This  occurs  in  both  Aretz  et  al.  (2006)  and  Donmez  et  al.  (2008), 
where  course  deviations  of  0  are  coded  with  no  tactor  activations. 

In  conclusion,  current  research  on  tactile  perception  provides  some  guidance  on  the  design  of 
layouts  for  tactile  vests.  For  our  designs,  inter-tactor  spacing  of  3  cm  (tactor  edge  to  tactor  edge) 


should  be  used.  In  addition,  duty  cycles  of  3  off:  1  on  could  be  used  to  prevent  adaptation.  Higher 
duty  cycles  could  still  be  used  if  they  are  not  on  for  longer  than  2  minutes.  Higher  duty  cycles 
may  also  lead  to  higher  perceived  intensity  due  to  temporal  summation,  and  may  be  appropriate 
for  high  urgency  situations. 

2.2  Proposed  Auditory  and  Tactile  Displays 

The  review  of  the  relevant  literature  on  multimodal  displays  in  UAV  simulators  demonstrates  that 
there  are  many  possibilities  for  implementing  auditory  and  tactile  displays  for  the  current  UAV 
autoland  scenario.  Multimodal  signals  could  provide  information  on  monitored  variables,  alerts, 
warnings,  or  they  could  help  provide  information  to  guide  the  operator’s  attention.  Through 
discussions  and  meetings  with  the  scientific  authority  and  with  a  current  civilian  pilot,  a  number 
of  proposed  displays  were  developed  for  displaying  attitude  upsets,  engine  RPM,  and  for 
displaying  various  warnings. 

2.2.1  Pattern  Displays  vs.  Symbolic  Displays 

For  the  tactile  displays  proposed,  it  may  be  possible  to  display  information  through  tactile 
symbols.  However,  little  research  has  been  conducted  on  how  efficiently  individuals  are  able  to 
differentiate  between  various  symbols  created  with  tactors  (Self,  Van  Eip,  Eriksson,  &  Elliott, 
2008).  Yanagida  et  al.  (2004)  demonstrated  that  letter  recognition  with  tactors  was  relatively 
successful  (87%  correct  letter  or  number  recognition).  However,  it  should  be  noted  Yanagida  et 
al.  (2004)  did  not  ask  participants  to  perform  any  additional  tasks.  In  high  workload  situations, 
where  a  user  is  forced  to  perform  additional  tasks,  these  results  may  not  hold  true  and  we  may 
observe  a  decline  in  accuracy. 

In  another  study  by  McKinley  et  al.  (2005),  three  different  tactors  were  used  to  present  enemy, 
unknown,  and  friendly  aircrafts  to  pilots.  The  location  of  the  tactors  on  the  vest  indicated  the 
spatial  location  of  the  target  aircrafts  relative  to  the  subject’s  aircraft.  The  results  of  this  study 
indicated  that  the  differentiation  between  the  various  tactors  was  not  an  easy  task.  Also,  the 
individual’s  familiarity  with  the  necessary  set  of  symbols  affected  accuracy  during  the  pattern 
recognition  process.  Thus,  the  size  of  the  symbol  set  and  the  training  given  to  operators  become 
critical  factors. 

Another  important  factor  is  the  type  of  data  that  is  being  conveyed.  In  our  baseline  GCS  interface, 
attitude  information  is  conveyed  continuously  through  the  UAV’s  camera  and  through  the  use  of 
visual  displays.  System  fault  information,  on  the  other  hand,  is  presented  in  discrete  units.  Due  to 
continuous  or  discrete  characteristics  of  the  information  being  conveyed,  tactile  patterns  would  be 
more  appropriate  for  continuous  information  such  as  attitude.  Tactile  symbols  would  be  better  at 
presenting  discrete  warning  information. 

2.2.2  Attitude  Upsets 

One  of  the  critical  variables  that  UAV  operators  are  responsible  for  monitoring  is  the  attitude  of 
the  UAV.  The  UAV’s  attitude  is  composed  of  its  roll,  pitch,  and  yaw.  Deviations  from  the  UAV’s 
proposed  attitude  can  lead  to  an  inability  for  the  UAV  to  land  safely.  The  tactile  vest  was  chosen 
as  the  primary  method  for  presenting  attitude  upset  information  to  the  UAV  operator.  Roll,  pitch, 
and  yaw  are  all  spatial  variables,  and  deviations  in  any  of  these  dimensions  could  easily  be  shown 
using  intuitive  spatial  representations  on  the  tactile  vest  in  a  manner  similar  to  that  used  by  Aretz 
et  al.  (2006).  However,  we  also  provide  suggestions  for  an  auditory  implementation  of  many  of 
the  tactile  displays  as  a  way  of  providing  redundant  or  supplementary  information. 


Four  different  types  of  displays  were  developed  to  show  attitude  upsets:  roll,  pitch,  yaw,  and  a 
combined  attitude  upset  measure.  For  the  individual  roll,  pitch,  and  yaw  displays,  it  was 
important  to  show  both  the  magnitude  of  the  deviation  and  the  direction  of  the  deviation.  For  the 
combined  attitude  upset  measure,  magnitude  was  determined  to  be  the  sole  factor  which  was 
important  to  display. 

As  mentioned  in  our  previous  literature  review  (Giang  et  al.  2010),  tactile  patterns  and 
perceptions  of  apparent  movement  can  be  generated  by  sequentially  activating  a  series  of 
vibrotactors  placed  on  the  skin  (Cholewiak  &  Collins,  2000).  Patterns  have  been  used  to 
intuitively  present  information  regarding  orientation  or  direction  of  external  events.  Given  the 
success  of  such  patterns,  we  feel  that  patterns  could  also  be  used  to  show  attitude  information  to 
UAV  controllers.  We  believe  using  tactile  patterns  should  reduce  the  amount  of  error  and 
processing  time  for  understanding  the  UAV  situation  by  the  operators. 

Another  important  issue  when  designing  for  attitude  upsets  is  whether  or  not  directionality  of  the 
deviation  is  necessary  information.  When  directionality  is  included,  the  user  is  provided  with 
additional  information  about  the  type  of  deviation  that  is  occurring,  and  may  be  able  to  better 
grasp  the  current  situation  of  the  UAV.  For  example,  a  UAV  that  is  experiencing  additional  roll  in 
the  direction  of  a  turn  is  probably  suffering  from  a  different  problem  than  a  UAV  that  is 
experiencing  roll  in  a  direction  that  is  opposite  that  of  the  intended  turn.  However,  directionality 
also  increases  the  complexity  of  the  tactile  and  auditory  signals.  This  may  result  in  patterns  that 
are  more  difficult  to  interpret  and  learn.  Also,  just  the  knowledge  that  attitude  upset  is  present 
may  result  in  the  participant  checking  the  visual  displays,  which  may  be  a  better  way  for 
communicating  detailed  information.  Due  to  these  issues,  we  felt  that  during  situations  where  we 
were  designing  for  more  detailed  displays  (i.e.,  separate  displays  for  attitude  upsets), 
directionality  was  still  an  important  characteristic  to  portray.  However,  for  aggregate  displays, 
directionality  was  less  important. 

2.2.2. 1  Proposed  Method  for  Displaying  Roll 

A  suggested  tactor  configuration  for  displaying  the  angle  of  the  UAV’s  roll  is  depicted  in  Figure 
2.4.  This  configuration  requires  21  tactors  and  uses  5  activated  tactors  per  level.  The  tactor 
configurations  can  be  positioned  either  on  the  front  or  back  of  the  vest. 
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Figure  2.4:  Suggested  tactor  configuration  for  roll  presentation. 
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This  tactor  configuration  presents  information  through  moving  tactile  patterns.  It  is  possible  to 
present  three  completely  separate  levels  of  roll  through  the  configuration  provided  above.  The 
mapping  of  degrees  of  roll  deviation  to  the  different  levels  is  a  task  that  must  be  accomplished  at 
a  later  stage.  The  first  level  of  roll  deviation  is  presented  by  activating  the  tactors  located  on  the 
green  dotted  line.  For  example  if  the  UAV’s  roll  deviation  is  less  than  20  degrees  to  the  right,  we 
can  show  this  by  activating  the  tactors  aligned  on  the  right  side  of  the  green  dotted  line,  indicated 
by  “1  R.”  Higher  levels  of  roll  deviation  can  be  shown  by  activating  the  tactors  aligned  on  the 
yellow  and  red  dotted  lines.  For  example  if  the  UAV’s  roll  deviation  is  more  than  20  degrees  and 
less  than  40  degrees  to  the  left,  we  can  show  this  by  simultaneously  activating  the  tactors  aligned 
with  the  yellow  dotted  lines  designated  “2  L.” 

2. 2. 2. 2  Tactor  Activation  Sequence  for  Displaying  Roll 

The  simultaneous  activation  of  two  vibrotactors  located  close  together  causes  the  sensation  of 
only  a  single  point  between  the  two  tactors  (apparent  location).  This  point  shifts  continuously 
toward  the  vibration  with  higher  intensity  (Scherrick,  Cholewiak,  &  Collins,  1990).  We  can  apply 
this  phenomenon  to  our  proposed  method,  to  remove  the  feeling  of  discrete  levels  of  UAV  roll 
deviation  on  the  tactor  vest.  For  example,  when  a  UAV’s  roll  changes  from  “1  R”  to  “2  R,”  as  in 
Figure  2.5,  the  tactor  activation  sequence  would  be  as  follows: 

1.  Reduce  the  vibration  amplitude  of  the  “1  R”  tactors  from  100%  to  50%  of  their  maximum 
value  and  increase  the  vibration  amplitude  of  the  “2  R”  tactors  from  0%  to  50%  of  their 
maximum  value. 

2.  Reduce  the  vibration  amplitude  of  the  “1  R”  tactors  from  50%  to  0%  of  their  maximum 
value  and  increase  the  vibration  amplitude  of  the  “2  R”  tactors  from  50%  to  100%  of 
their  maximum  value. 

By  using  this  activation  strategy  we  can  provide  sensations  of  apparent  locations  between  the 
actual  locations  of  the  tactors.  This  may  aid  in  decreasing  the  confusion  in  understanding  the 
UAV’s  current  roll  deviation  by  providing  more  perceivable  levels  of  roll  information. 
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Figure  2.5:  Tactor  activation  sequence  for  roll  display. 
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2. 2. 2. 3  Auditory  Display  for  Presenting  Roll 


Our  auditory  roll  signal  is  very  similar  to  the  one  we  proposed  for  the  tactile  roll  signal.  The  main 
goal  of  the  roll  display  (for  both  the  auditory  and  tactile  displays)  is  to  show  rapid  changes  in 
attitude,  especially  ones  that  oscillate.  To  accomplish  this  we  need  to  show  both  the  magnitude 
and  frequency  of  the  changes.  In  tactile  displays,  frequency  information  is  limited  by  how  quickly 
we  can  change  the  tactile  patterns  to  show  the  oscillations  and  how  well  the  operators  can  detect 
these  changes.  During  high  turbulence  (as  might  be  experienced  during  wind  shear),  these 
changes  may  be  beyond  what  our  tactor  display  is  able  to  display.  Thus,  we  propose  that  a 
redundant  auditory  signal  be  used  in  conjunction  with  the  tactile  display. 

The  auditory  signal  will  provide  more  detailed  information  about  the  frequency  of  roll  changes 
(how  often  the  roll  has  switched  directions).  Our  auditory  signal  will  provide  a  beep  every  time 
the  roll  changes  directions.  During  very  turbulent  periods  the  rate  of  the  auditory  beeps  will 
provide  a  better  measure  of  the  severity  of  the  turbulence.  This  information  will  be  synergistic 
with  the  magnitude  (and  low  resolution  frequency)  changes  shown  using  our  tactor  display. 

However,  we  can  also  modify  the  auditory  component  to  show  magnitude  information  by 
changing  the  intensity  (loudness)  of  the  auditory  signal  as  the  magnitude  increases.  Because  roll 
is  a  continuous  variable,  the  changes  in  intensity  will  also  be  continuous.  However,  3D  auditory 
displays  may  not  be  practical  for  implementation  for  this  study.  Thus,  we  decided  that  the 
direction  of  the  roll  should  be  represented  by  the  changing  the  ear  to  which  the  auditory  signal  is 
presented  (using  stereo  headphones).  Roll  to  the  left  would  be  signalled  with  presentation  to  the 
left  ear  and  roll  to  the  right  would  be  signalled  with  presentation  to  the  right  ear. 

2. 2. 2. 4  Proposed  Method  for  Displaying  Yaw 

The  effectiveness  of  a  vibrotactile  torso  display  as  a  countermeasure  to  spatial  disorientation  was 
investigated  by  Van  Erp  et  al.  (2006).  In  this  study,  participants  wore  a  vibrotactile  display  vest 
which  consisted  of  24  columns  of  2  vibrotactors  while  seated  on  a  rotating  chair.  This  vibrotactile 
display  was  designed  to  help  participants  recover  from  spatial  disorientation.  The  results  of  this 
experiment  demonstrated  that  a  vibrotactile  display  can  assist  operators  as  they  recover  from  loss 
of  spatial  orientation. 

Inspired  by  the  Van  Eip  et  al.  (2006)  experiments,  our  display  indicates  the  UAV’s  yaw  (heading) 
deviation  using  a  simple  tactor  configuration.  As  shown  in  Figure  2.6,  a  row  of  tactors,  consisting 
of  five  C2  tactors,  is  placed  in  the  frontal  region  of  the  tactor  vest.  The  UAV  heading  can  be 
displayed  by  sequentially  activating  the  vibrotactors  along  the  observer’s  chest  or  abdomen  along 
the  horizontal  plane.  In  this  method,  the  vibrotactile  signal  moves  in  the  same  direction  of  the 
UAV’s  yaw  deviation.  The  tactor  located  on  the  midsagittal  plane  of  the  torso  acts  as  the  null 
point  in  this  configuration. 

It  should  be  noted  that  this  display  is  only  activated  when  the  UAV’s  heading  deviates  from  its 
desired  heading.  For  example,  if  the  UAV  is  programmed  to  travel  from  Point  A  to  Point  B,  it  has 
to  hold  a  specific  heading.  If  for  any  reason  (such  as  turbulence  or  wind-shear)  the  UAV’s 
heading  deviates  from  its  desired  value,  the  yaw  display  can  be  activated  to  show  the  intensity  of 
the  yaw  divergence.  This  is  illustrated  in  Figure  2.7. 


Figure  2.6:  Suggested  factor  configuration  for  yaw  presentation. 


Desired  heading 


2. 2.2. 5  Proposed  Method  for  Displaying  Pitch 

Our  suggested  configuration  for  displaying  pitch  is  very  similar  to  the  configuration  used  in  the 
yaw  display.  As  can  be  seen  in  Figure  2.8,  this  configuration  makes  use  of  a  vertical  column  of 
five  C2  tactors  in  the  frontal  region  of  the  tactor  vest.  The  UAV  pitch  can  be  displayed  by 
sequentially  activating  the  vibrotactors  aligned  along  the  observer’s  midsagittal  plane.  In  this 
method,  the  vibrotactile  stimulus  moves  in  the  same  direction  as  the  UAV’s  pitch  deviation.  The 
middle  tactor  (the  third  of  five  tactors)  acts  as  the  null  point  in  this  configuration. 


Midsagittal  plane 


A  C2  tactor 


Transverse  plane 


Figure  2.8:  Suggested  tactor  configuration  for  pitch  presentation. 

Similar  to  the  yaw  display,  this  display  is  only  activated  when  the  UAV’s  pitch  angle  deviates 
from  its  desired  pitch  angle.  For  example,  during  a  UAV  recovery,  the  UAV  may  need  to  increase 
or  decrease  its  pitch  to  land  safely.  As  shown  in  Figure  2.9,  if  for  any  reason  (such  as  turbulence 
or  wind  shear)  the  UAV’s  pitch  angle  deviates  from  its  desired  value,  the  pitch  display  can  be 
activated  to  show  the  intensity  and  direction  of  the  pitch  angle  divergence. 


Desired  pitch  angle 


UAV's  pitch  angle 
Pitch  angle  divergence 
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Figure  2.9:  UAV's  pitch  angle  and  desired  pitch  angle. 


2. 2. 2. 6  Combination  of  Yaw  and  Pitch  Displays  to  form  Yaw-Pitch  Display 

In  the  last  few  sections  we  introduced  and  described  the  relevant  tactor  configurations  to  present 
yaw  and  pitch  through  the  tactor  vest.  We  believe  that  we  can  combine  these  configurations  to 
construct  a  more  complex  tactor  display.  We  call  it  a  yaw-pitch  display. 

As  Figure  2.10  shows,  the  yaw-pitch  display  would  be  constructed  of  a  5x5  tactor  matrix.  This 
tactor  configuration  is  capable  of  showing  yaw  and  pitch  intensity  concurrently.  The  intensity  of 
yaw  can  be  presented  by  activation  of  tactors  along  the  horizontal  axis  and  the  intensity  of  pitch 
can  be  displayed  by  activation  of  tactors  along  the  vertical  axis.  Therefore  the  yaw  intensity 
would  be  presented  in  terms  of  the  distance  between  the  activated  tactor  and  the  midsagittal  plane 


of  the  body,  and  the  pitch  angle  would  be  presented  in  terms  of  the  distance  between  the  activated 
tactor  and  the  transverse  plane  of  the  torso. 


Figure  2.10:  Suggested  tactor  formation  for  pitch  and  yaw  display. 

For  instance,  if  the  tactor  indicated  in  red  is  activated  (shown  in  Figure  2.10),  this  would  represent 
a  situation  in  which  the  UAV  has  a  yaw  deviation  to  the  right  and  downwards  pitch  deviation. 

The  use  of  two  tactile  patterns  in  combination  is  one  that  has  not  been  reported  in  the  literature 
and  this  combination  may  reveal  many  questions  regarding  how  efficiently  operators  can  detect 
the  two  patterns  concurrently.  We  feel  that  the  patterns  provide  information  which  is  sufficiently 
different  and  intuitive,  that  the  operator  would  be  able  to  make  use  of  both  displays  at  once.  This 
should,  however,  be  investigated  further. 

2. 2.2. 7  Aggregate  Attitude  Upset  Display 

In  the  previous  sections  we  explored  attitude  displays  that  would  provide  information  about  flight 
dynamic  parameters  that  could  be  presented  independently  and  concurrently.  The  roll  display  was 
designed  such  that  it  could  be  mounted  on  the  back  of  the  vest,  while  the  pitch-yaw  displays 
could  be  mounted  on  the  front  side  of  the  vest.  Therefore,  it  may  have  been  possible  for  the  UAV 
operator  to  perceive  the  intensity  of  the  roll,  pitch,  and  yaw  using  the  as  individual  indicators  of 
flight  dynamics  though  the  vibrotactile  vest. 

However,  there  are  other  methods  for  presenting  the  magnitude  of  the  UAV’s  deviation  from  its 
desired  direction  using  the  tactile  vest.  One  possible  method  is  to  use  the  vibrotactile  display  to 
show  the  general  deviation  of  the  UAV  from  its  desired  track.  Using  this  method  the  operator 
would  not  be  able  to  differentiate  between  the  intensity  of  the  yaw,  pitch,  and  roll  deviations. 
However,  they  would  be  presented  as  an  aggregate  indicator  of  attitude  upset.  Therefore,  in  this 
type  of  display  the  UAV  operator  will  only  have  a  general  perception  of  the  UAV  spatial 
orientation.  Such  a  display  may  provide  enough  information  to  the  UAV  operator  to  allow  them 
to  make  an  informed  judgement  about  whether  the  UAV  is  able  to  land. 

The  following  are  two  proposed  methods  for  conveying  this  aggregate  attitude  indicator. 


2. 2.2. 7.1 


Column  of  Tactors 


The  suggested  tactor  configuration  for  a  column  of  tactors  display  is  depicted  in  Figure  2.1 1.  As 
the  figure  illustrates,  the  configuration  of  the  vibrotactors  in  this  display  is  very  simple.  Eight  C2 
tactors  are  placed  in  a  vertical  row  along  the  midsagittal  plane  of  either  the  front  or  back  of  the 
vest  to  form  a  column. 


(a) 


Figure  2.11:  Column  of  Tactors 


(b) 


Studies  have  revealed  that  spine  and  the  navel  can  work  as  natural  anchor  points  and  observers 
are  more  capable  of  correctly  detecting  stimuli  near  these  points  (Cholewiak  et  al.,  2004;  Van 
Erp,  2005).  Therefore,  the  tactor  column  should  be  placed  in  such  a  way  that  it  aligns  to  the 
observer’s  midsagittal  plane.  Based  on  the  activation  sequence  of  the  C2  tactors,  there  are  two 
methods  for  showing  the  aggregate  attitude  upset  using  a  column  of  tactors. 


Activation  in  Sequence 

In  this  method  of  activation,  the  vertical  distance  of  the  activated  tactor  from  the  body’s  pelvis 
plane  (the  bottom  of  the  torso)  represents  the  magnitude  of  the  attitude  upset.  For  example  when 
the  magnitude  of  the  UAV’s  deviation  changes  from  Level  1  to  Level  2,  the  Level  1  tactor  is 
deactivated  and  the  Level  2  tactor  is  activated.  This  would  correspond  to  an  increase  in  the 
UAV’s  attitude  upset.  One  possible  weakness  of  this  display  is  the  ability  for  UAV  operators  to 
discriminate  between  the  different  levels  of  the  display.  The  operators  may  not  be  able  to  clearly 
and  easily  recognize  the  exact  location  of  the  vibration  on  their  torso  when  there  is  only  one  tactor 
activated. 


Equalizer  activation 

In  this  type  of  activation,  the  number  of  activated  tactors  and  the  location  of  the  upper-most 
activated  tactor  represent  the  magnitude  of  the  deviation.  Using  this  method,  the  tactor  located  on 
the  lower  level  is  not  deactivated  when  the  attitude  upset  increases  from  one  level  to  another.  For 
example,  when  the  UAV’s  attitude  upset  is  at  Level  3,  all  of  the  tactors  on  lower  levels  (Level  1 
and  Level  2)  are  also  activated.  Similarly,  when  the  attitude  upset  increases  to  the  next  level,  all 


the  tactors  up  to  that  level  will  stay  activated.  One  metaphor  for  this  type  of  tactor  display 
functionality  would  be  the  graphical  equalizer  display  in  electronic  audio  devices.  The  length  of 
the  vibrating  column  represents  the  intensity  of  the  UAV’s  deviation  from  its  desired  path.  One 
advantage  of  this  type  of  activation  is  that  operators  are  not  relying  solely  on  the  location  of  the 
vibrating  tactor  because  the  number  of  tactors  vibrating  acts  as  a  redundant  cue.  This  could  also 
potentially  convey  some  degree  of  urgency  as  the  net  tactile  stimulus  increases.  One  disadvantage 
would  be  possible  adaptation  or  annoyance  because  some  tactors  (the  lower  levels  ones 
especially)  would  be  on  for  much  longer  durations  as  the  UAV’s  attitude  upset  increases. 

2. 2.2. 7. 2  Uni-centred  Circles  of  Tactors 

The  second  proposed  display  for  showing  an  aggregate  attitude  upset  can  be  seen  in  Figure 
2. 12. In  this  configuration  tactors  are  located  in  uni-centred  circles. 


Figure  2.12:  Tactor  formation  for  the  uni-centred  circles  of  tactors. 

In  this  type  of  display,  the  tactors  which  are  located  equidistant  from  the  center  of  the  circle  are 
activated  simultaneously  and  represent  the  various  levels  of  the  display.  The  size  of  the  circles 
represents  the  magnitude  of  the  deviation  of  the  UAV  from  its  desired  path.  The  larger  the  circle 
is,  the  higher  the  represented  attitude  upset  is.  This  design  uses  the  location  of  the  tactors 
(represented  by  the  distance  of  the  tactors  from  the  center)  and  the  number  of  tactors  as  ways  of 
conveying  magnitude  information.  Other  possible  variations  of  this  design  are  square  arrays  of 
tactors  or  other  shapes. 

It  is  worth  noting  that  the  magnitude  information  presented  may  not  seem  to  increase  linearly 
because  the  number  of  tactors  activated  is  not  increased  at  a  linear  rate,  but  rather  as  a  function  of 
area  covered  by  the  tactors  used.  There  is  currently  little  research  literature  to  indicate  whether 
this  method  of  increase  would  result  in  a  perceptual  effect  of  a  squared  magnitude  or  a  near  linear 
increase  in  magnitude.  We  know  that  in  a  visual  display,  biases  can  be  introduced  when  linear 
variables  are  mapped  to  non-linear  graphics  (Tufte,  1997  provides  good  examples  of  this).  This 
effect  may  not  be  as  evident  in  tactile  displays,  due  to  the  reduced  and  varying  sensitivity  of  the 
tactile  display  zone  on  the  body.  If  the  effect  occurs  however,  the  display  may  need  to  be 
modified  place  the  different  levels  more  appropriately,  based  on  the  attitude  upset. 

2. 2.2. 8  Advantages  and  Disadvantages  of  Aggregate  Attitude  Displays  vs. 
Separate  Attitude  Displays 

In  this  section  we  will  briefly  discuss  the  various  benefits  and  drawbacks  of  the  ideas  presented 
above,  compared  to  those  which  would  allow  the  presentation  of  individual  attitude  upsets.  Both 


of  these  methods  could  serve  a  role  in  alerting  the  operator  to  re-examining  the  visual  interface  in 
off-nominal  situations.  The  choice  of  a  single  solution  would  be  highly  dependent  on  the  training, 
experimental  tasks,  and  kinds  of  task  switching  behaviour  we  expect  from  our  participants. 

One  of  the  strongest  advantages  for  the  use  of  an  aggregate  attitude  upset  display  is  the  simplicity. 
One  danger  of  tactile  displays,  and  multimodal  interfaces  in  general,  is  overloading  the  operator 
with  information.  During  the  2010  Human  Factors  and  Ergonomics  Society  Annual  Meeting,  the 
discussion  panel  on  tactile  displays  emphasized  that  the  bandwidth  of  tactile  displays  is  limited, 
and  thus,  simple  and  important  messages  may  be  best  suited  for  tactile  displays.  An  aggregate 
attitude  upset  display  would  alert  the  operator  to  any  deviation  to  the  UAV’s  flight  path,  while  not 
requiring  the  operator  to  integrate  different  pieces  of  information  to  come  to  the  conclusion  that 
the  UAV  has  veered  off  its  intended  path.  Thus,  this  type  of  display  would  alert  the  operator  to 
glance  back  at  the  visual  displays  and  gather  further  information. 

There  are  two  main  disadvantages  for  the  use  of  the  aggregate  display.  The  first  is  the  need  to 
have  some  sort  of  formulation  to  generate  a  single  value  for  attitude  upset  based  on  roll,  yaw,  and 
pitch  information.  In  most  aircraft,  attitude  indicators  (such  as 
http://en.wikipedia.org/wiki/Attitude  indicator)  show  pitch  and  roll  as  two  separate  dimensions 
on  a  display.  Heading  is  shown  with  other  instruments.  From  our  research,  there  are  no  standard 
methods  of  mapping  the  three  attitude  measures  into  a  single  aggregate  value.  While  there  are 
some  examples  from  attitude  control  systems,  the  equations  are  relatively  complex  due  to  the 
need  to  combine  three  angular  measures  compared  against  a  dynamic  frame  of  reference.  If  an 
aggregate  display  were  chosen  for  implementation,  we  would  need  to  formulate  our  own 
mathematical  solution  to  calculate  attitude.  This  method  could  be  advantageous  because  we  could 
then  design  a  formula  that  would  be  most  indicative  of  the  types  of  phenomenon  we  are  designing 
our  display  to  be  sensitive  to  (wind  shear  and  turbulence).  A  simple  example  of  a  formula  would 
be  to  normalize  each  of  the  separate  attitude  upset  measures  (roll,  pitch,  and  yaw)  and  to  sum 
them  into  one  variable.  However,  any  formulation  will  require  some  degree  of  fine  tuning  and 
require  exploration  of  various  attitude  equations  to  develop  the  most  effective  representation  for  a 
human  operator.  Some  generalization  of  the  end  results  would  be  lost  as  our  experimental  results 
would  be  dependent  on  a  unique  computation  of  the  general  attitude  variable.  The  argument  could 
also  be  made  that  our  computed  equation  introduced  bias  by  making  certain  parameters  of  our 
experimental  task  more  salient. 

The  second  major  disadvantage  of  an  aggregate  display  would  be  the  lack  of  situational 
awareness  information  that  the  tactile  display  would  provide.  Depending  on  the  type  of  formula 
used,  many  different  inputs  of  pitch,  yaw,  and  roll  would  provide  the  same  aggregate  value. 
Operators  would  not  be  able  to  differentiate  between  these  values.  In  addition,  because  the 
aggregate  display  would  be  less  informative  of  flight  dynamics  overall,  it  may  provide  very  little 
benefit  over  the  use  of  an  attitude  upset  alarm.  Operators  may  choose  to  glance  at  the  visual 
display  anytime  the  tactile  display  was  activated.  This  may  occur  regardless  of  the  degree  of 
severity  of  the  attitude  upset  because  the  tactile  display  itself  does  not  provide  any  further 
information.  Separate  attitude  displays,  on  the  other  hand,  may  provide  operators  with  greater 
levels  of  information.  Greater  information  could  allow  the  operator  to  delay  switching  away  from 
a  secondary  task  if  they  deem  it  is  unnecessary  due  to  the  additional  situation  awareness  created 
by  the  separate  displays. 

In  the  case  of  separate  attitude  displays,  the  display  can  be  simplified  by  choosing  to  only  use  one 
of  the  attitude  dimensions  (e.g.  pitch  or  roll).  The  variable  remains  ecologically  compatible  with 


the  flight  dynamics  of  the  UAV  and  is  a  variable  that  pilots  understand  well  from  their  training. 
While  the  display  still  needs  to  be  tuned  to  set  the  mapping  correctly,  the  variable  should  be 
mapped  in  such  a  way  that  it  requires  no  computation.  The  directional  information  available  in 
the  display  allows  the  pilot  to  extract  more  information  than  simply  magnitude,  which  should 
improve  situation  awareness. 

2.2.3  Warnings  and  Alerts 

Other  possible  variables  that  could  be  displayed  using  auditory  and  tactile  stimuli  are  the 
warnings  and  alerts  that  are  currently  shown  visually  on  the  UAV  interface.  These  alerts  cover  a 
wide  variety  of  situations  that  the  UAV  may  encounter.  Some  are  non-critical  and  provide 
information  such  as  “landing  gear  deployed,”  and  some  are  critical  and  require  immediate 
response  such  as  “engine  failure.”  We  developed  two  proposed  warnings  and  alerts  for  the 
multimodal  interface.  Both  of  these  displays  can  be  presented  through  either  auditory  or  tactile 
stimuli,  and  presentation  methods  in  both  these  modalities  are  described  below.  As  suggested  by 
prior  work  (Donmez,  Graham,  &  Cummings,  2008),  continuous  displays  are  better  at  presenting 
continuous  information  and  threshold  displays  are  better  are  presenting  discrete  data.  Therefore, 
the  use  of  warnings  and  alerts  are  better  for  providing  awareness  of  discrete  events.  Thus,  it  is  our 
proposal  that  the  multimodal  alerts  be  used  for  monitoring  system  faults  that  are  currently 
represented  using  discrete  visual  alerts  in  the  baseline  GCS. 

2.2.3. 1  Simple  Alert 

Our  first  proposed  solution  is  to  map  each  of  the  system  fault  warnings  (e.g.,  high  engine 
revolutions  per  minute  (RPM)  warnings,  low  engine  RPM  warnings,  or  high  engine  temperature 
warnings)  to  a  simple  auditory  or  tactile  alarm.  The  purpose  of  this  alert  would  be  to  direct  the 
operator’s  visual  attention  back  to  the  GCS  interface  so  that  they  can  gather  further  information. 
Operators  can  make  use  of  this  information  to  minimize  the  amount  of  time  spent  monitoring  the 
GCS  interface  by  allowing  for  the  alert  to  prompt  when  they  should  shift  their  attention.  This,  in 
turn,  should  allow  the  operator  to  spend  more  time  on  the  secondary  task,  leading  to  higher 
accuracy  and  faster  response  times. 

From  the  information  currently  available  about  system  faults  in  the  simulator,  we  did  not  think 
that  different  alarm  priorities  were  required.  Thus,  we  envision  a  simple  alert  that  will  only  sound 
once  (when  the  GCS  first  presents  the  visual  alert).  The  auditory  alert  would  consist  of  a  short 
duration  (~0.2s)  beep,  similar  to  the  discrete  alerts  used  in  Graham  and  Cummings  (2007).  The 
tactile  alerts  would  make  use  of  the  tactors  placed  on  the  thighs  and  they  would  also  vibrate  for 
only  a  short  duration  (~0.2s).  The  intensities  of  the  auditory  and  tactile  signals  would  be  constant 
throughout  the  duration  of  the  alert. 

Findings  from  Calhoun  et  al.  (2004)  suggest  that  both  auditory  and  tactile  redundant  cues  would 
be  effective  at  improving  performance  over  just  a  visual  cue.  Therefore,  we  think  that  either  of 
the  cues  could  work  and  the  best  solution  would  be  dependent  on  other  multimodal  stimuli.  In  our 
experiment,  the  faults  are  always  reliable,  hence  optimal  behaviour  would  be  to  only  inspect  the 
GCS  interface  when  alerted  by  the  auditory  and  tactile  alert  (or  when  the  operator  feels  like  there 
are  detrimental  environmental  conditions  which  are  not  presented  through  these  alerts).  Thus,  an 
operator  will  reach  optimal  performance  when  they  become  completely  reliant  on  these  alerts  for 
keeping  track  of  system  faults. 


2. 2. 3. 2  Complex  User-Aware  Alert 


Our  second  proposed  alert  is  one  that  will  make  use  of  an  eye-tracking  system  to  gain  some 
awareness  of  what  the  operator  is  currently  doing.  In  our  experiment,  the  workload  of  the 
secondary  task  is  constant,  therefore  there  is  little  opportunity  to  switch  between  tasks  (i.e. 
switching  during  low  workload  periods  and  ignoring  alerts  during  high  workload  periods). 
However,  it  may  be  prudent  to  design  an  alert  system  which  can  be  ignored  by  the  operator  when 
they  prioritize  higher  workload  tasks.  Thus,  we  propose  an  alert  that  will  slowly  increase  in 
intensity  from  when  the  fault  first  occurs,  until  the  operator  finally  deals  with  the  fault  (either 
through  the  concern  or  abort  buttons).  Whenever  the  operator  glances  at  the  GCS  console,  the 
intensity  of  the  alert  will  be  reset  to  the  starting  point,  but  will  continue  to  grow  until  the  situation 
is  resolved.  We  have  decided  to  use  the  temporal  rate  of  the  alert  as  a  method  for  manipulating 
the  intensity.  This  is  because  the  speed  of  an  alarm  was  found  to  have  the  greatest  effect  on  the 
perceived  urgency  of  an  auditory  stimulus  (Hellier  &  Edworthy,  1999). 

The  auditory  alert  will  consist  of  a  short  duration  (~0.1s)  beep,  which  will  increase  in  rate  as  the 
duration  of  the  alert  increases.  The  alert  will  have  an  initial  rate  of  1  repetition  per  second  and 
will  reach  a  maximum  intensity  of  8  repetitions  per  second  within  10  seconds  from  the  beginning 
of  the  alert.  These  values  may  need  to  be  further  refined  after  pilot  testing.  The  tactile  alert  will 
follow  a  similar  pattern.  If  the  eye-tracker  detects  that  the  operator  has  looked  at  the  GCS 
interface,  then  the  intensity  of  the  alert  will  be  reset  to  its  initial  state.  The  advantage  of  using  a 
display  such  as  this  is  that  it  allows  the  operator  to  prioritize  their  task  switching  while  staying 
aware  of  how  long  they  have  ignored  the  primary  task. 

2.2.4  Engine  RPM 

In  the  current  GCS  interface,  high  and  low  engine  RPM  are  shown  as  discrete  warnings  through  a 
visual  display.  However,  the  use  of  auditory  and  tactile  displays  allows  us  to  show  this  previously 
discrete  variable  as  a  continuous  variable.  The  sound  of  an  engine’s  RPM  is  something  that  many 
people  naturally  use  as  an  indicator  of  the  engine’s  health.  Thus,  it  is  a  good  candidate  variable 
for  monitoring  using  sonification.  The  goals  of  the  RPM  sonification  would  be  an  ambient  sound 
that  would  alert  the  operator  to  low  and  high  RPM  events  without  becoming  overly  distracting. 

2.2.4. 1  Related  Sonifications 

A  quick  review  of  the  literature  showed  no  examples  where  RPM  has  been  sonified.  Some 
reasons  for  this  may  be  because  engine  RPM  already  has  a  strong  auditory  component  that  many 
individuals  recognize.  However,  in  a  UAV,  where  the  operator  is  removed  from  the  actual 
vehicle,  an  auditory  representation  of  RPM  would  be  necessary.  There  are  similar  examples  of 
sonifications  which  have  been  used  to  convey  aircraft  information.  One  example  is  a  sonification 
designed  by  Watson,  Sanderson,  and  Anderson  (2000)  that  described  a  complex  sonification  for 
aircraft  landing.  In  this  sonification  used  a  mapping  involving  the  3-D  location  of  the  sound, 
order,  pitch,  timbre,  and  tempo.  The  3-D  location  of  the  sound  represented  the  direction  of 
deviation  from  the  approach  path.  Airspeed  was  mapped  onto  the  tempo  of  the  sonification.  The 
throttle  and  the  mode  of  flight  were  mapped  onto  the  pitch  and  the  timbre  of  the  sonification. 
Finally,  the  different  engines  could  be  differentiated  by  the  order  of  the  sounds.  This  sonification 
is  quite  complex  and  makes  use  of  multiple  characteristics  of  the  carrier  signal.  A  sonification  for 
engine  RPM  could  be  much  simpler  and  due  to  simplicity  it  may  be  possible  to  also  provide 
additional  redundancy  by  using  multiple  characteristics  to  represent  the  same  changes. 


2. 2.4. 2  Potential  Mappings  for  RPM 


The  following  are  a  few  proposed  methods  for  mapping  engine  RPM  onto  auditory 
characteristics.  These  auditory  characteristics  can  be  applied  to  a  repeating  auditory  carrier  signal. 
The  ranges  for  these  mappings  will  need  to  be  further  explored  once  the  ranges  for  engine  RPM 
are  known,  but  these  mappings  should  provide  some  ideas  about  how  we  can  represent  the  data  as 
an  auditory  signal.  A  final  sonification  combining  all  of  these  mappings  may  be  the  best  method 
for  creating  such  a  sonification. 

2. 2.4. 2.1  Pitch  Mapping  of  RPM 

The  first  method  for  mapping  RPM  data  to  a  sound  is  to  have  RPM  mapped  onto  pitch.  This 
would  be  a  simple  mapping,  with  high  pitch  representing  high  engine  RPM  and  low  pitch 
representing  low  engine  RPM.  While  pitch  may  not  be  the  most  influential  auditory  component 
for  conveying  auditory  urgency  (Hellier  &  Edworthy,  1999),  it  is  a  highly  recognizable  auditory 
characteristic  that  should  be  easily  interpreted  by  most  operators. 

2. 2.4. 2. 2  Tempo  Mapping  of  RPM 

Another  possible  method  for  mapping  RPM  data  to  sound  is  to  change  the  tempo  as  the  RPM 
changes.  This  is  intuitive  because  higher  RPM  normally  correlates  with  faster  tempos  of  sounds 
coming  from  the  engine.  Similarly,  low  RPM  would  map  intuitively  onto  a  slow  tempo.  Speed  is 
a  very  influential  parameter  for  auditory  urgency  (Hellier  &  Edworthy,  1999),  thus  high  engine 
RPM  sounds  will  probably  be  perceived  as  highly  urgent.  Low  engine  RPM  on  the  other  hand 
may  not  be  given  the  same  advantages  in  urgency,  as  low  engine  RPM  essentially  generates  “less 
noise.”  This  could  be  counter-intuitive  as  a  low  engine  RPM  situation  may  be  as,  or  more,  critical 
than  a  high  engine  RPM  situation.  However,  the  mapping  is  ecologically  valid  as  in  an  actual 
turboprop  driven  aircraft  (such  as  the  Heron),  lower  engine  RPM  would  result  in  a  lower  tempo  of 
the  propellers.  In  the  case  of  trained  pilots,  this  may  be  a  condition  they  are  sensitive  to  and  low 
tempos  may  retain  a  sense  of  urgency. 

2.2. 4.2. 3  Timbre  Mapping  of  RPM 

Another  way  of  showing  high  and  low  engine  RPM  is  by  alerting  the  operator  to  when  the  engine 
RPM  deviates  from  the  norm.  This  can  be  represented  through  the  use  of  the  sonification  of 
timbre.  At  high  and  low  engine  RPM,  engines  tend  to  start  producing  sounds  that  are  not  heard 
during  normal  operation.  This  can  be  represented  in  the  sonification  by  changing  the  quality  of 
the  sound.  During  normal  RPM  values,  the  sound  of  the  sonification  can  be  “noisier”  or 
“messier.”  Hopefully,  this  will  lead  to  a  sound  that  is  less  intrusive  and  one  that  sounds  similar  to 
a  UAV  engine’s  normal  operating  sound.  As  engine  RPM  deviates  from  the  norm  or  average 
RPM,  the  sound  can  become  crisper  and  clearer.  This  may  cause  the  sound  to  become  more 
noticeable  to  the  operator. 

2.2. 4.2. 4  Combined  Mappings 

The  various  mappings  could  be  combined  so  that  RPM  was  mapped  to  pitch,  tempo,  and 
potentially  timbre.  The  additional  redundancy  provided  by  the  combined  mapping  may  increase 
the  salience  of  the  display.  With  the  mappings  combined,  the  sonification  would  be  as  follows: 

Normal  RPM:  noisy,  mid  pitch,  mid  tempo 


High  RPM:  clear,  high  pitch,  high  tempo 
Low  RPM:  clear,  low  pitch,  low  tempo. 

2. 2.4. 2. 5  Attentional  and  Urgency  Mapping  for  Engine  RPM 

Another  important  concept  to  consider  when  designing  the  auditory  sonification  is  the  urgency 
mapping  that  would  best  support  the  monitoring  task.  Different  auditory  characteristics  lead  to 
different  subjective  perceptions  of  urgency  (Hellier  &  Edworthy,  1999).  This  is  a  concept  that  can 
be  used  to  create  alarms  of  different  priorities  or  to  indicate  situations  that  require  more 
immediate  responses  in  an  intuitive  manner  (Arrabito  et  al,  2004).  A  similar  concept  which  is 
applied  more  directly  to  sonifications  is  the  idea  of  attentional  mapping  (Sanderson  et  al.,  2000). 
Attentional  mapping  is  an  extension  to  Ecological  Interface  Design,  where  explicit  mappings  for 
where  attention  should  be  directed  during  different  system  states  is  designed.  The  goal  is  to  create 
shifts  of  attention  from  peripheral  to  focal  awareness  when  a  signal  moves  from  a  normal  state  to 
an  abnormal  state.  For  monitoring  engine  RPM  there  are  two  abnormal  states  and  one  normal 
state.  Both  low  and  high  RPM  indicate  a  problem  has  occurred,  with  low  RPM  being  more  urgent 
than  high  RPM.  Normal  engine  RPM,  on  the  other  hand,  should  provide  the  user  with  ambient 
feedback  without  being  salient. 

Attentional  and  urgency  mapping  can  differ  from  the  simple  coding  for  engine  RPM  into 
variables  such  as  pitch,  tempo,  and  timbre.  For  example,  it  is  simple  and  intuitive  to  map  the 
speed  to  the  engine  to  pitch,  however,  high  pitch  is  perceived  as  more  urgent  than  low  pitch 
(Hellier  &  Edworthy,  1999).  Thus,  even  though  low  pitch  can  be  easily  interpreted  as 
representing  low  engine  RPM,  the  urgency  mapping  would  be  inappropriate.  Ideally,  the  engine 
RPM  sonification  would  be  composed  of  two  components:  the  information  content  (the  engine 
RPM)  and  how  critical  the  information  is  (the  urgency).  To  achieve  this  we  propose  a  simple 
saliency  mapping  that  uses  both  tempo  and  pitch.  Pitch  would  still  be  used  to  indicate  the 
information  content  of  the  sonification  (i.e.,  the  engine  RPM  with  low  pitch  representing  low 
engine  RPM  and  high  pitch  representing  high  engine  RPM)  and  with  tempo  representing  the 
saliency  mapping  component  (with  low  tempo  representing  normal  situations  and  high  tempo 
representing  high  urgency  situations).  To  create  the  correct  attentional  mapping,  it  would  be 
important  to  have  a  very  low  saliency  tempo  used  for  normal  engine  RPM  which  would  increase 
gradually  as  the  engine  RPM  either  rose  or  fell.  It  would  still  be  important  to  ensure  that  the 
tempo  mapping  for  low  engine  RPM  would  increase  at  a  faster  rate  than  the  tempo  mapping  for 
high  engine  RPM  to  retain  the  greater  urgency  that  low  engine  RPM  represents.  Further 
experimentation  to  understand  these  mappings  would  still  be  required. 

The  proposed  attentional  and  urgency  mappings  are  as  follows: 

Normal  RPM:  mid  pitch,  slow  tempo 

High  RPM:  high  pitch,  high  tempo 

Low  RPM:  low  pitch,  highest  tempo 


2.3  Conclusion 


In  this  chapter  we  examined  the  current  literature  on  auditory  and  tactile  design  in  UAV 
simulation  environments  and  provided  descriptions  of  a  number  of  different  auditory  and  tactile 
designs  which  can  be  used  to  increase  operator  awareness  during  UAV  autoland  scenarios.  One  of 
the  lessons  learned  from  the  process  of  generating  the  auditory  and  tactile  designs  was  that  the 
design  space  for  this  problem  is  very  large.  There  are  a  number  of  different  variables  that  can  be 
important  to  the  monitoring  of  an  automated  UAV.  The  variables  explored  in  this  chapter  include 
attitude  upsets,  system  alerts  and  warnings,  and  engine  RPM.  While  any  of  these  variables  may 
increase  operator  awareness,  the  task  of  finding  which  variables  are  most  efficient  at  conveying 
information  on  the  UAV’s  ability  to  land  was  beyond  the  scope  of  this  current  project.  Even  for 
the  few  variables  discussed  in  this  chapter,  there  are  numerous  methods  for  coding  variables  into 
the  auditory  and  tactile  display  spaces.  Where  possible,  we  drew  from  previous  literature  to 
support  our  designs. 

One  issue  that  was  especially  difficult  to  judge  in  our  designs  was  how  much  information  is 
required  to  provide  additional  operator  awareness.  For  example,  individual  roll,  pitch,  and  yaw 
displays  would  provide  the  operator  with  a  better  overall  understanding  of  the  UAV’s  current 
situation.  However,  the  additional  information  provided  over  an  aggregate  display  may  not  be 
required  for  an  operator  whose  sole  decision  is  whether  to  abort  or  follow  through  with  the  UAV 
recovery.  Due  to  the  limited  options  available  to  the  operator,  a  simpler  display  may  actually 
prove  more  useful.  Through  discussions  with  the  scientific  authority,  two  displays  were  chosen 
for  further  development.  The  aggregate  attitude  tactile  vest  display  and  the  engine  RPM 
sonification  were  refined  and  implemented  in  simple  prototypes  for  testing.  The  following  chapter 
describes  the  process  of  implementation  and  the  simple  experiment  designed  to  test  the 
capabilities  of  these  auditory  and  tactile  displays. 


3  Auditory  and  Tactile  Display  Prototyping  and 
Testing 


In  this  chapter  we  focus  on  the  development  and  prototyping  of  a  tactile  and  auditory  display  first 
proposed  in  Sections  2.2.2  and  2.2.4.  An  aggregate  attitude  upset  tactile  display  and  an  engine 
RPM  sonification  were  chosen  for  further  refinement  and  development.  Our  goal  was  to  explore 
how  to  implement  these  displays  using  the  hardware  provided  and  to  develop  a  simple  testing 
methodology  for  evaluating  how  participants  would  respond  to  the  different  interface  ideas. 

3.1  Auditory  Stimuli  Generation 

In  this  section  of  the  report,  the  generation  and  coding  of  the  auditory  stimuli  are  discussed.  In 
particular,  we  developed  three  different  kinds  of  sounds.  The  first  was  based  on  a  real  aircraft 
propeller  engine.  The  second  was  generated  using  the  same  aircraft  sound  but  with  changes  in  the 
tempo  of  the  sound.  Finally,  we  created  a  sonification  using  pure  tones.  A  sonification  is  used  to 
present  a  data  variable  through  a  continuous  auditory  signal,  in  this  case  the  auditory  signals  used 
were  a  propeller  sound  signal  and  a  pure  tone  signal.  The  propeller  sound  is  more  realistic,  but 
more  complex  to  generate.  The  propeller  sound  with  tempo  is  even  more  complex,  but  allows  us 
to  increase  the  amount  of  information  conveyed.  The  pure  tone  based  sound  is  less  realistic  but 
much  simpler  to  generate.  The  following  sections  explain  the  steps  that  occurred  to  generate  the 
auditory  stimuli. 

3.1.1  Generating  Signals 

A  Tucker  Davis  RP2. 1  Real-Time  audio  processor  (Tucker  Davis  Technologies,  2011;  shown  in 
Figure  3.1)  was  the  audio  signal  generator  and  acquisition  hardware  used  for  generating  the  signals 
for  each  of  our  auditory  displays. 
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Figure  3.1:  Tucker  Davis  RP2.1  Real-Time  audio  processor.  (Image  taken  from  Tucker  Davis 

Technologies,  2011). 

The  RP2.1  audio  processor  can  be  controlled  through  software  provided  by  Tucker  Davis  Real- 
Time  Processor  Visual  Design  Studio  (RPvdsEX;  Tucker  Davis  Technologies,  2011).  This 
software  has  a  visual  design  environment  that  is  flexible,  easy  to  use,  and  allows  users  to  create 
custom  signal  processing  programs  without  writing  complex  code.  The  programming 
environment  provides  different  signal  processing  components  in  the  form  of  blocks  which  can  be 
selected  and  placed  simply  to  create  custom  audio  processing  circuits.  Audio  circuits  can  then  be 


programmed  into  the  RP2.1  audio  processor.  In  the  following  sections  we  will  describe  the 
auditory  circuits  used  to  generate  the  different  auditory  displays. 

3.1.2  Generating  Propeller  Engine  Sounds 

For  generating  a  complex  signal  like  a  propeller  engine  sound,  we  need  to  know  the  constituent 
frequencies  of  that  signal.  We  also  need  to  know  the  amplitudes  of  the  individual  notes  involved 
in  the  complex  signal.  A  Fourier  transform  is  a  mathematical  operation  that  can  identify  the 
constituent  frequencies  and  their  amplitudes.  Essentially,  a  Fourier  transform  decomposes  a 
complex  signal  into  simpler  oscillatory  signals. 

The  sound  of  a  real  propeller  engine,  working  at  normal  RPM,  was  recorded  in  the  form  of  a 
WAV  file.  This  file  was  provided  by  Defence  Research  and  Development  Canada  (DRDC) 
Ottawa.  Fourier  transform  operations  were  performed  by  inserting  the  WAV  file  into  a  computer 
program  designed  to  execute  the  desired  Fourier  transforms.  This  program  was  written  using  the 
MATLAB  programming  language. 


Figure  3.2:  Results  of  the  Fourier  transform  of  the  propeller  engine  sound. 

According  to  the  Fourier  transform  results  which  are  depicted  in  Figure  3.2,  the  propeller  engine 
sound  is  a  signal  which  can  be  structured  by  summing  up  a  series  of  sine  wave  signals  at  107.2, 
214.3,  320.7,  429.1,  and  583.5  Hz  with  the  magnitudes  of  2.94,  1.22,  0.79,  0.09,  and  0.04 
respectively.  The  fundamental  frequency  is  107.2Hz  and  the  other  frequencies  are  approximately 
the  multiples  of  the  fundamental  frequency.  It  should  be  noted  that  there  is  no  dimension  for  the 
magnitude  of  the  constituent  signals.  The  vertical  axis  of  the  Fourier  transform  results  represent 
the  ratio  of  magnitudes  of  the  constituent  signals  in  regards  to  each  other.  The  design  objective 
was  to  create  a  sound  with  a  similar  frequency  profile  to  this  sound. 

Figure  3.3  illustrates  the  audio  circuit  that  we  designed  to  generate  the  propeller  engine  sound.  As 
can  be  seen  in  Figure  3.3,  the  circuit  is  comprised  of  8  blocks.  In  the  following  sections,  the 
blocks  are  described  in  more  detail. 
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Figure  3.3:  Screen  shot  of  the  RPvdxEX  software.  The  audio  circuit  for  generating  the  propeller 

engine  sound. 


3.1. 2.1  Tone 

Tone  is  a  circuit  component  which  generates  a  sinusoid  waveform  with  a  specific  frequency  and 
amplitude  (Tucker  Davis  Technologies,  2011).  As  mentioned  previously,  a  propeller  engine 
sound  can  be  generated  by  adding  a  series  of  sinusoid  signals  at  specific  frequency  and  amplitude 
levels.  We  used  5  blocks  of  Tone  in  our  design  in  order  to  provide  the  sine  waves  required  to 
produce  the  propeller  engine  sound.  Each  tone  block  provides  one  of  the  fundamental  frequencies 
that  were  identified  in  the  Fourier  transform  of  the  original  signal.  As  well,  the  amplitude  of  each 
frequency  component  has  been  reconstructed  in  each  Tone  block. 

3.1 .2.2  Sum 

Sum  is  a  multi-input  component  that  performs  basic  summation.  It  adds  all  of  the  input  signals 
and  provides  the  result  at  the  time  of  output  (Tucker  Davis  Technologies,  2011).  As  can  be  seen 
in  Figure  3.3,  all  of  the  propeller  engine  constituent  signals  are  given  to  the  Sum  as  inputs.  The 
resulting  output  is  the  desired  engine  sound. 


3.1 .2.3  DacOut  (CH=1 ,  CH=2) 


A  DacOut  function  block  can  be  summarized  as  digital  to  analog  conversion  and  provides  a 
signal  at  the  designated  output  channel  (Tucker  Davis  Technologies,  2011).  Sending  a  signal  to  a 
DacOut  block  allows  users  to  listen  to  the  generated  signals  using  the  proper  headphones  or 
speakers.  The  RP2.1  processor  has  2  possible  output  channels  (Two  possible  output  channels 
allow  for  different  signals  to  different  ears  or  to  different  output  devices).  In  this  case,  the 
propeller  signal  is  being  output  on  both  audio  outputs. 

3.1.3  Propeller  Engine  Sound  with  Increasing/Decreasing  RPM 

The  propeller  engine  sound  discussed  in  Section  3.1.2  provides  an  auditory  signal  which  presents 
the  sound  of  a  propeller  engine  under  normal  working  conditions  and  at  a  normal  RPM  level. 
However,  the  proposed  auditory  display  requires  a  design  where  sound  varies  at  different  RPM 
levels.  Therefore,  we  decided  to  increase  or  decrease  the  constituent  frequencies  in  a  linear 
manner  to  map  different  RPM  levels.  Figure  3.4  illustrates  an  auditory  circuit  which  was  designed 
in  the  RPvdsEX  software  in  order  to  generate  a  propeller  engine  sound  with  increasing  RPM. 

Considering  Figure  3.4,  A  LinRamp  is  one  of  the  auditory  circuit  components  in  RPvdsEX 
software  which  generates  a  linear  ramp  that  begins  at  the  min  value  and  goes  to  the  max  value 
over  a  specified  time  (Tucker  Davis  Technologies,  2011).  As  we  mentioned  previously,  the 
fundamental  frequency  of  the  propeller  engine  sound  is  107.2Hz  and  the  other  constituent 
frequencies  are  approximately  the  multiples  of  the  fundamental  frequency.  In  order  to  simplify 
the  design,  we  decided  to  feed  the  frequency  of  the  Tone  blocks  from  a  single  LinRamp  source.  In 
this  case  the  LinRamp  constantly  sets  the  frequency  of  the  Tone  blocks  over  the  time. 

As  can  be  seen  in  Figure  3.4  the  min  value  of  the  LinRamp  is  107.2,  which  is  equal  to  the 
fundamental  frequency  of  the  propeller  engine  sound.  In  order  to  construct  other  constituent 
frequencies  we  used  Divide  blocks  at  the  output  of  the  LinRamp.  A  Divide  block  divides  the  input 
by  the  denominator  and  passes  the  quotient  to  the  output  (Tucker  Davis  Technologies,  2011). 

In  the  auditory  circuit  depicted  in  Figure  3.4  the  ramp  time  is  8000ms.  If  the  ramp  time  decreases, 
the  increment  rate  of  the  LinRamp  increases.  This  results  in  the  generation  of  a  propeller  engine 
sound  which  increases  in  pitch  over  time,  representing  an  increase  in  RPM  over  time. 
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Figure  3.4:  The  audio  circuit  for  generating  the  propeller  engine  sound  with  increasing  RPM. 

3.1.4  Adding  Differing  Tempo  Rates  to  the  Propeller  Engine  Sound  as  a 
Way  of  Displaying  Different  Levels  of  Urgency 

As  mentioned  in  the  previous  section,  we  can  generate  a  propeller  engine  sound  which  varies 
RPM  levels.  In  a  real  propeller  engine,  however,  low  and  high  RPM  conditions  are  more  critical 
than  normal  RPM  conditions.  In  these  cases  we  wanted  to  map  urgency  along  with  the  propeller 
sound.  In  order  to  do  this,  we  chose  to  control  the  tempo  rate  of  the  output  signals,  as  a  redundant 
factor,  to  show  different  levels  of  urgency  (Edworthy  et  al.,  1991).  Controlling  the  tempo  means 
activating  and  deactivating  the  output  signals  at  different  timing  intervals.  The  timing  of 
activation  and  deactivation  needed  to  be  a  function  of  the  changes  in  RPM  levels. 

As  can  be  seen  in  Figure  3.5,  the  designed  auditory  circuit  was  comprised  of  two  main  parts.  Part 
I  and  Part  II.  Part  I  is  depicted  with  a  red  boundary  and  Part  II  is  depicted  with  a  blue  boundary. 
In  this  auditory  circuit  the  higher  RPM  urgency  level  is  mapped  onto  faster  tempo  rates.  The 
functionality  of  this  circuit  is  such  that  the  RPM  level  and  the  tempo  rate  of  the  generated  sound 
starts  at  a  constant  level  and  remains  steady  for  the  first  10  seconds  (Part  I).  After  10  seconds,  the 
RPM  and  the  tempo  rates  start  to  increase  linearly  for  20  seconds  (Part  II). 


Figure  3.5:  Adding  different  tempo  rates  to  the  propeller  engine  sound  as  a  way  of  displaying 

different  levels  of  urgency. 

As  can  be  seen  in  Figure  3.5,  a  combination  of  a  Schmitt  block  and  a  NOT  gate  component 
(designated  with  the  green  circles)  is  included  in  the  design  of  the  proposed  circuit,  for  switching 
purposes.  A  Schmitt  block  output  goes  to  a  high  state  for  a  set  amount  of  time  (determined  by 
Thi).  At  the  end  of  the  high  time,  the  output  goes  to  a  low  state  for  a  set  time  (determined  by  Tlo). 
The  combination  of  the  Schmitt  block  and  the  Not  gate  here  works  like  a  digital  switch.  They  stop 
Part  1  of  the  circuit  from  transferring  data  to  the  output  after  1 0  seconds  and  allow  Part  II  of  the 
circuit  to  transfer  data  to  the  output  at  the  same  time.  We  have  provided  detailed  descriptions 
about  each  part  in  the  following  sub  sections. 


3.1. 4.1  Parti 


The  major  structure  of  the  Part  I  circuit  (depicted  with  a  purple  boundary  in  Figure  3.5)  is  very 
similar  to  the  circuit  which  was  illustrated  in  Figure  3.3.  This  part  of  the  circuit  generates  a 
propeller  engine  sound  which  is  working  at  a  specific  RPM  level.  The  RPM  level  can  be  set  by 
changing  the  fundamental  frequency  of  the  circuit  through  the  ConstF  block.  In  general  a  ConstF 
block  provides  a  specific  constant  which  can  be  inputted  to  any  other  block  as  an  input. 

This  design  gives  us  the  capability  to  set  the  fundamental  frequency  more  easily.  For  instance,  if 
the  desired  starting  fundamental  frequency  is  150Hz,  we  can  change  the  value  of  the  ConstF 
block  to  150Hz.  The  other  multiples  of  the  fundamental  frequencies  will  be  generated 
automatically  through  the  Divide  blocks  and  will  be  inputted  to  the  corresponding  Tone  blocks. 

The  resulting  output  of  Part  I  of  the  circuit  is  a  propeller  engine  sound  which  works  at  a  specific 
RPM  level  (designated  through  the  ConstF  block  value)  with  a  tempo  rate  of  500ms  on  and 
500ms  off.  The  on/off  action  is  carried  out  by  modulating  the  output  of  circuit  in  the  purple 
boundary  through  a  subsequent  modulating  circuit  (depicted  in  Figure  3.5  with  a  brown  circle). 
The  modulating  circuit  is  comprised  of  a  PulseTrain  and  a  Cos2Gate  block.  A  PulseTrain  block 
generates  logic  pulse  trains  and  turns  the  Cos2Gate  block  on  and  off  at  each  500ms  time  interval. 

A  Cos2Gate  block  functions  as  an  enable  line.  The  Cos2Gate  will  stay  on  and  transfers  data  when 
its  CTRL  parameter  value  is  HI.  The  rise  time  is  the  time  it  takes  for  the  signal  to  reach  90%  of 
the  maximum  value.  The  fall  time  is  the  time  it  takes  the  signal  to  reach  10%  of  the  maximum 
value.  The  signal  will  start  to  decrease  in  frequency  as  soon  as  the  CTRL  value  is  low. 

3.1. 4.2  Part  II 

The  resulting  output  of  Part  II  of  the  circuit  is  a  propeller  engine  sound  with  an  increasing  RPM 
level  which  causes  the  tempo  of  the  generated  signal  to  increase.  As  can  be  seen  from  Figure  3.5 
the  combination  of  LinRamp,  Divide,  ConstF,  and  PulseTrain  blocks  constructed  the  modulating 
circuit  for  this  part  (depicted  with  a  yellow  circle).  The  functionality  of  this  part  of  the  circuit  is 
very  similar  to  the  modulating  section  of  the  Part  I  circuit.  The  difference  here  is  that  the  on/off 
time  linearly  decreases  over  time. 

3.1.5  Generating  Pure  Tone  Sounds 

The  alternative  to  generating  a  realistic  propeller  sound  was  to  generate  a  sonification  built  from 
pure  tones. 

Figure  3.6  illustrates  an  auditory  circuit  which  generates  a  pure  tone.  As  can  be  seen  in 

Figure  3.6,  it  is  much  easier  to  generate  a  sinusoid  signal  using  the  RP2.1  processor.  It  can  be 
done  by  using  only  a  Tone  block  and  sending  the  output  to  a  DacOut. 


Figure  3.6:  Generating  a  sine  wave  at  250Hz  through  RPvdsEX  software. 

It  is  possible  to  increase  or  decrease  the  frequency  of  a  pure  tone  by  adding  additional  blocks 
which  will  change  the  frequency  that  is  inputted  into  the  tone.  Figure  3.7  illustrates  an  auditory 
circuit  which  generates  a  sine  wave  signal  with  decreasing  frequency.  The  frequency  of  the 
generated  signal  starts  at  250Hz  and  decreases  to  0Hz  over  10  seconds.  Similarly,  an  increase  in 
frequency  can  be  achieved  using  a  linear  ramp  that  is  inputted  into  the  tone  generator. 


Figure  3. 7:  Generating  a  sine  wave  decreasing  in  frequency. 

3.2  Attitude  Upset  Tactile  Vest  Implementation 

In  Section  2.2.2  we  proposed  some  methods  for  displaying  the  attitude  of  an  UAV  using  the 
vibrotactile  vest.  We  suggest  two  main  configurations.  In  the  first  configuration  the  C2  tactors 
formed  a  column  of  tactors  and  in  the  second  configuration  tactors  were  located  in  form  of  uni- 
centred  circles. 

Generally,  as  we  move  from  distal  regions  (such  as  the  hands)  to  proximal  regions  (such  as  the 
torso)  of  the  body,  the  sensitivity  to  stimuli  degrades.  The  law  of  mobility  states  that  the  skin’s 
sensitivity  to  locating  and  discriminating  touched  locations  improves  as  the  mobility  of  parts  of 
the  body  increases  (Cholewiak,  Brill,  &  Schwab,  2004;  Van  Eip,  2005b).  Hence,  displaying 
information  through  the  uni-centred  circle  formation  might  be  confusing  for  subjects.  In  addition, 
many  of  the  more  complex  solutions  discussed  in  Section  2  (such  as  the  uni-centered  circle 
formation,  the  yaw  and  pitch  displays)  require  multiple  tactors  located  on  the  front  or  back  of  a 
vest.  The  arrangement  of  some  of  these  designs  may  not  be  possible  given  the  recommended  3  cm 
tactor-to-tactor  separation,  and  to  maximize  discriminability  a  simpler  tactor  layout  would  be 
required.  After  some  discussions  with  DRDC  and  the  research  group,  we  concluded  that  using  the 
column  of  tactors  method  would  be  a  more  effective  solution  for  displaying  information  through 
the  tactile  vest.  However,  using  one  vertical  column  of  tactors  on  the  back  or  front  side  of  the 
vest,  aligned  the  observer’s  navel  or  spine,  would  be  inefficient.  The  tactor  vest  does  not  have 


good  contact  with  the  body  surfaces  in  line  with  the  spine  or  navel.  Therefore,  we  decided  to  use 
a  column  of  tactors  to  the  left  or  right  side  of  the  spinal  column  in  order  to  provide  the  necessary 
contact. 

Decreasing  the  number  of  tactors  can  increase  a  participant’s  ability  to  discriminate  between  the 
different  locations.  As  can  be  seen  in  Figure  3.8  some  minor  changes  were  made  in  the  primary 
designs.  We  decided  to  reduce  the  number  of  tactors  in  a  column  from  eight  to  four.  This  was 
done  to  ensure  that  each  tactor  would  have  sufficient  spacing  to  increase  detectability  and 
discrimination.  We  also  decided  to  position  the  tactors  in  the  upper  region  of  the  back  to  ensure 
that  the  tactors  on  the  vest  would  have  contact  with  the  skin.  We  found  that  even  with  a  correctly 
fitted  vest,  tactors  located  in  the  lower  region  of  the  back  would  sometimes  lose  contact  with  the 
skin  when  the  participant  was  seated.  This  may  have  been  due  to  improper  posture  and  slouching. 
We  suggest  that  the  tactors  be  placed  at  least  3  cm  apart  (edge-to-edge  distance)  as  suggested  by 
Van  Eip  (2005).  The  minimum  separation  of  3  cm  was  due  to  the  spatial  acuity  of  the  torso,  and 
tactors  which  are  located  closer  than  3  cm  apart  may  feel  like  they  are  originating  from  the  same 
tactor.  The  entire  four  (or  eight)  tactor  array  will  require  2 1  cm  (9  cm  for  the  inter-tactor  spacing 
and  12  cm  for  the  four  tactors).  From  the  NASA  anthropometric  data  set,  a  5th  percentile  40  year 
old  Japanese  female  has  a  waist-back  measurement  of  35.2  cm,  so  this  display  would  take  up 
roughly  60%  of  the  participant’s  back.  Similarly,  the  95th  percentile  40  year  old  American  male 
has  a  waist-back  measurement  of  5 1  cm,  which  results  in  the  display  taking  up  just  slightly  over 
40%  of  the  participant’s  back.  While  we  believe  that  these  values  are  acceptable,  future  designs 
may  require  scaling  of  the  inter-tactor  distances  to  match  variations  in  torso  size,  thus  reducing 
the  number  of  tactors  used  or  the  amount  of  space  on  the  back  used  may  need  to  be  adjusted 

It  is  also  possible  to  reduce  the  inter-tactor  spacing  to  2cm,  which  is  the  lower  bound  of  spatial 
acuity  suggested  by  Van  Erp  (2005).  However,  this  would  only  reduce  the  entire  tactor 
configuration  to  18  cm.  This  would  still  result  in  a  display  that  would  take  up  roughly  50%  of  the 
participant’s  back  for  a  5th  percentile  40  year  old  Japanese  female  and  roughly  35%  of  the 
participant’s  back  for  a  95th  percentile  40  year  old  American  male.  Further  investigation  is  needed 
to  see  if  the  loss  in  spatial  acuity  is  outweighed  by  the  benefits  of  having  better  skin  contact  due 
to  the  reduced  size. 


Figure  3.8:  Reduced  the  number  of  tactors  from  eight  to  four  and  located  them  on  the  left  or  right 

side  of  the  spinal  column. 


Based  on  the  timing  properties  of  the  signals  of  the  tactors,  there  are  three  main  ways  of  attitude 
upset  display  using  the  column  of  tactors  configuration.  We  have  provided  detailed  descriptions 
about  these  methods  in  the  following  sections. 


It  should  be  noted  that  all  of  the  described  methods  in  this  section  can  also  be  executed  using  two 
columns  of  tactors,  one  column  on  the  left  and  one  column  on  the  right  side  of  the  spinal  column. 
The  purpose  of  using  two  columns  of  tactors  would  be  to  ensure  that  subjects  receive  the 
vibrations  if  a  miss-contact  happens  in  one  of  the  columns.  However  the  activation  sequence 
would  be  identical  for  both  columns. 

3.2.1  Activation  in  Sequence 

In  this  method  of  activation,  the  vertical  distance  of  the  activated  tactor  from  the  body’s 
transverse  plane  presents  the  magnitude  of  the  deviation.  For  example  when  the  magnitude  of  the 
UAV’s  deviation  changes  from  Level  1  to  Level  2  (see  Figure  3.8),  the  Level  1  tactor  is 
deactivated  and  the  Level  2  tactor  is  activated. 

A  C2  tactor’s  signal  may  vary  in  timing  parameters.  For  example,  it  is  possible  to  activate  a  tactor 
using  signals  which  are  different  in  terms  of  duty  cycle  or  period.  We  have  proposed  some 
strategies  for  displaying  information  by  using  different  duty  cycles  of  the  signals. 

3.2. 1.1  Activating  all  of  the  Tactors  using  Signals  with  a  Same  Duty  Cycle 

The  signal  is  identical  for  the  all  four  levels  of  tactors  in  this  method.  All  of  the  tactors  will  be 
activated  through  a  same  signal  at  the  time  of  activation.  As  can  be  seen  in  Figure  3.9,  the  period 
of  the  signal  is  250ms  and  the  duty  cycle  is  20%. 
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Figure  3.9:  Signal  with  the  period  of 250  ms  and  the  duty  cycle  of  20  %>. 

3. 2. 1.2  Activating  the  Tactors  using  Signals  with  Different  Duty  Cycles 

In  this  method  of  activation,  the  signal  for  each  level  of  tactors  is  different  from  the  other  levels 
(refer  to  Figure  3.10).  The  period  of  the  signals  are  identical  for  all  levels  but  the  duty  cycle 
increases  as  the  level  of  the  activated  tactor  increases.  For  example  if  the  magnitude  of  the  UAV’s 
deviation  is  at  the  lowest  level,  the  Level  1  tactor  will  be  activated  with  a  signal  with  a  duty  cycle 
at  20%.  If  the  magnitude  of  the  deviation  increases,  the  Level  1  tactor  is  deactivated  and  the  Level 
2  tactor  is  activated  by  applying  a  signal  with  a  duty  cycle  at  40%.  In  this  method  the  duty  cycle 
of  the  signal  for  the  Level  3  and  the  Level  4  tactors  are  60%  and  80%  respectively. 


Figure  3.10:  Activating  the  factors  using  signals  with  different  duty  cycles. 

3.2.2  Drawing  Line  of  Tactors 

Spatial-temporal  patterns  and  perceptions  of  apparent  movement  can  be  generated  by  sequentially 
activating  a  series  of  vibrotactors  placed  on  the  skin  (Cholewiak,  &  Collins,  2000).  In  this  section 
we  propose  an  activation  method  in  which  the  magnitude  of  the  UAV’s  deviations  are  presented 
in  terms  of  linear  patterns. 

In  this  method  the  tactors  are  used  to  draw  a  line  on  the  skin.  The  perceived  length  of  the  line 
presents  the  magnitude  of  the  deviations.  For  example  when  the  magnitude  of  the  UAV’s 
deviation  is  at  Level  2,  the  Level  1  and  Level  2  tactors  will  be  sequentially  activated.  If  the 
magnitude  of  the  deviation  increases,  the  Level  3  tactor  will  be  added  to  the  sequence.  The 
activation  timing  is  such  that  the  on-time  of  the  signals  of  the  tactors  located  on  two  consecutive 
levels  will  have  20ms  overlap.  Figure  3.11  depicts  the  signals  timings  for  the  case  in  which  all 
four  tactors  are  being  used  to  draw  the  tactile  line. 
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Figure  3.11:  Sequentially  activating  tactors  to  generate  a  tactile  line.  The  on-time  of  the  signals 
of  the  tactors  located  on  two  consecutive  levels  has  20  ms  overlap. 


As  can  be  seen  from  Figure  3.11,  the  signal  is  identical  for  all  four  levels  of  tactors  in  terms  of 
period  and  duty  cycle. 

3.2.3  Equalizer  Activation 

As  we  mentioned  previously,  with  this  type  of  activation  the  number  of  activated  tactors 
represents  the  magnitude  of  the  deviation.  In  this  mode,  the  tactor  located  on  the  lower  level  is  not 
deactivated  when  the  track  deviation  increases  from  one  level  to  another. 

3.2.3. 1  Activating  all  of  the  Tactors  using  Signals  with  the  Same  Duty  Cycle 

Similar  to  in  Section  1.1,  the  signal  is  identical  for  all  four  levels  of  tactors  in  this  method.  All  of 
the  tactors  will  be  activated  with  the  same  signal  at  the  time  of  activation.  The  period  of  the  signal 
is  250ms  and  the  duty  cycle  is  20%.  The  difference  here  is  that  when  the  attitude  upset  increases 
from  one  level  to  the  next,  all  the  tactors  up  to  that  level  will  stay  activated. 

3. 2. 3. 2  Activating  the  Tactors  using  Signals  with  Different  Duty  Cycles 

Similar  to  Section  3.2. 1.2,  in  this  method  of  activation  the  signal  for  each  level  of  tactors  is 
different  from  the  other  levels.  The  period  of  the  signals  are  identical  for  all  levels  but  the  duty 
cycle  increases  as  the  level  of  the  activated  tactor  increases.  The  duty  cycle  of  the  signal  for 
Levels  1  through  4  are  20%,  40%,  60%,  and  80%  respectively.  Again,  in  the  equalizer  activation 
mode,  the  tactor  located  on  a  lower  level  is  not  deactivated  when  the  track  deviation  increases 
from  one  level  to  another. 


3.3  Testing  of  Auditory  and  Tactile  Designs 

One  of  the  goals  of  implementing  the  auditory  and  tactile  designs  first  proposed  in  Section  2  was 
to  subject  the  displays  to  short  usability  tests.  As  mentioned  previously,  the  auditory  and  tactile 
design  space  is  very  large.  Even  with  the  initial  design  proposals,  each  design  still  had  many 
possible  design  variations  that  required  further  testing.  The  usability  test  would  be  attempt  to 
compare  and  contrast  different  variations  of  auditory  and  tactile  displays  and  to  examine  how 
well  they  perform  in  terms  of  discriminability  and  urgency.  It  is  important  to  note  that  these 
usability  tests  will  help  us  gain  an  understanding  about  the  different  characteristics  of  our  design, 
and  the  designs  tested  may  not  represent  the  final  multimodal  interfaces  which  will  be  used  in  the 
GCS.  Instead,  the  results  of  these  usability  studies  will  be  used  to  refine  and  augments  the  designs 
for  the  next  phase  of  the  GCS  project. 

3.3.1  Engine  RPM  Sonification 

The  auditory  design  chosen  for  further  testing  was  the  engine  RPM  sonification.  The  purpose  of 
the  sonification  is  to  provide  the  GCS  operator  information  about  the  UAV’s  current  engine  RPM 
value  and  to  cue  the  operator  to  off-nominal  situations  when  the  engine  RPM  value  is  either  too 
high  or  too  low.  The  current  baseline  GCS  interface  makes  use  of  visual  warnings  and  a  visual 
RPM  value  display  to  indicate  whenever  high  or  low  engine  RPM  occurs.  Flowever,  the  visual 
warnings  are  discrete,  and  while  they  are  highly  salient  when  the  operator  is  paying  attention  to 
the  interface,  and  they  do  not  provide  the  operator  with  any  sort  of  situation  awareness  of  a 
developing  situation.  The  visual  RPM  value  display  does  provide  the  operator  with  some 
awareness  of  the  health  of  the  situation,  but  this  display  is  a  small  component  of  the  display  and 
operators  may  not  be  paying  attention  to  the  value. 


In  Section  3.1  we  describe  the  implementation  of  three  different  types  of  displays  based  on  the 
auditory  characteristics  of  pitch,  tempo,  and  timbre.  In  our  first  design  we  make  use  of  a  propeller 
engine  like  sound  with  the  pitch  of  its  fundamental  frequency  mapped  onto  the  engine  RPM.  We 
used  a  mapping  of  higher  pitch  to  represent  higher  engine  RPM.  In  our  second  design,  we  used  a 
complex  propeller  like  sound  which  was  augmented  with  a  tempo  mapping.  As  the  engine  RPM 
increases,  so  does  the  tempo  of  the  sonification.  The  final  design  used  a  simple  pure-tone  which 
also  mapped  pitch  to  engine  RPM.  These  three  designs  represent  variations  in  terms  of  two 
factors:  the  base  carrier  signal  (propeller  like  sound  vs.  pure  tone)  and  whether  tempo  is  used  as 
an  additional  auditory  characteristic  (tempo  change  vs.  no  tempo  change).  These  four  designs 
represent  some  of  the  possible  displays  which  we  may  test  in  our  study.  Further  refinement  is  still 
required  to  narrow  down  the  parameters  and  the  displays  which  will  be  tested.  In  our  usability  test 
we  will  examine  and  compare  these  interfaces  based  on  discriminability  and  urgency. 

3. 3. 1.1  Discriminability  of  Auditory  Sonifications 

One  question  that  is  important  to  answer  with  regards  to  the  sonifications  is  whether  participants 
are  able  to  easily  discriminate  between  different  levels  of  engine  RPM  on  a  perceptual  level.  The 
auditory  designs  will  be  tested  using  a  “just  noticeable  difference  paradigm”.  Up  to  5  different 
levels  will  be  pre-selected  out  of  the  range  of  the  sonification  display.  During  each  trial, 
participants  will  experience  one  of  the  RPM  levels  as  a  stable  auditory  pattern  for  a  short  period 
of  time,  after  which  the  sound  or  tactile  pattern  will  either  increase  or  decrease.  Participants  will 
be  asked  to  indicate  when  they  have  noticed  the  pattern  has  changed  by  pressing  a  button  and  to 
indicate  the  direction  of  the  change  (increase  or  decrease).  This  will  stop  that  trial.  The  pattern 
will  be  repeated  at  new  level.  All  participants  will  experience  repetitions  of  each  trial  for  each 
design  in  a  within  subjects  design.  In  half  of  the  trials,  the  signal  will  increase  and  in  10  of  the 
trials  the  signal  will  decrease.  Increasing  and  decreasing  of  the  signals  will  be  randomized.  Two 
times  of  onset  of  the  change  of  the  signal  will  occur,  randomized  so  that  the  time  of  onset  of  the 
change  of  the  signal  cannot  be  anticipated  a  priori.  The  dependent  variables  will  be  response  time, 
which  is  calculated  from  the  start  of  the  increase/decrease  until  the  point  where  the  participant 
responds,  and  the  percentage  of  correct  responses. 

The  main  goal  of  the  discriminability  portion  of  the  experiment  is  to  examine  differences  between 
the  two  carrier  signals  (the  propeller  sound  and  the  pure-tone).  Since  the  signal  will  be 
continuous,  we  are  interested  in  examining  how  the  two  different  carrier  signals  would  impact  the 
participant’s  ability  to  detect  changes.  We  are  also  interested  in  whether  discriminability  is 
affected  by  different  starting  pitches.  If  higher  pitch  signals  produce  sonifications  that  are  easier 
to  discriminate  then  we  may  shift  up  the  range  of  frequencies  used. 

3. 3. 1.2  Urgency  of  Auditory  Sonifications 

During  situations  with  high  and  low  engine  RPM,  the  operator  is  required  to  take  action  to  ensure 
the  safe  recovery  of  the  UAV.  Thus,  it  is  important  that  the  sonification  effectively  alerts  the 
operator  to  these  situations.  By  encoding  a  sense  of  urgency  into  our  sonifications,  we  can 
intuitively  convey  this  type  of  information.  Assessing  the  perceived  urgency  of  the  auditory 
displays  will  help  us  evaluate  if  this  is  being  accomplished  with  our  proposed  designs. 

To  test  the  perceived  urgency  of  the  sonifications,  participants  will  be  presented  with  5  different 
engine  RPM  levels  repeated  a  number  of  times  at  each  level  and  will  be  asked  to  provide  a  rating 
of  urgency  Each  exposure  will  be  quite  short,  lasting  only  about  5  seconds.  The  rating  will  be 


provided  on  an  urgency  mapping  scale  based  on  Edworthy’s  urgency  mapping  principle  and 
described  in  Arrabito  et  al.  (2004).  This  scale  is  a  simple  scale  between  1  and  100.  The  order  of 
presentation  of  the  levels  will  be  randomized.  The  dependent  variable  will  be  the  rating  of 
perceived  urgency. 

3.3.2  Attitude  Upset  Tactile  Display 

A  similar  experiment  will  also  be  conducted  for  the  attitude  upset  tactile  display.  The  goal  of  the 
experiment  will  be  to  determine  the  tactor  layout  and  activation  pattern  that  will  provide  the  best 
discriminability  performance  and  provide  the  highest  degree  of  urgency  when  required.  The 
design  of  the  experiment  has  still  not  been  finalized.  One  factor  that  makes  the  evaluation  of 
discriminability  and  urgency  different  from  the  auditory  sonification  is  the  discreteness  of  the 
tactile  display.  Instead  of  presenting  a  continuous  signal,  the  tactile  display  uses  discrete  tactors 
that  are  only  able  to  show  levels  of  attitude  upset.  This  requires  different  methods  for  evaluation 
of  discriminability  performance  and  may  also  change  how  urgency  is  rated  by  participants. 

3.3.3  Experimental  Parameters 

There  are  still  a  number  of  experimental  parameters  that  must  be  defined  during  the  pilot  study 
for  this  experiment.  The  following  tables  list  each  of  the  parameters  that  must  be  set  for  the 
discriminability  task  and  the  urgency  task. 


Table  2:  Experimental  parameters  to  be  determined. 


Parameter 

Purpose 

Number  of  pre-set  levels. 

These  levels  act  as  starting  points  for  the  signal  to  either 
increase  or  decrease  from.  It  is  important  to  cover  the  entire 
range  of  the  display  because  the  mappings  used  may  have 
different  levels  of  discriminability  or  urgency  at  different 
levels.  For  example,  it  may  be  easier  to  detect  changes  at  high 
pitch  than  at  low  pitch. 

Stimulus  Onset 

Asynchrony  (SOA) 

SOA  refers  to  the  amount  of  time  before  the  stimuli  changes. 
The  SOA  varies  so  that  the  participant  does  not  anticipate 
when  a  change  would  occur.  We  suspect  that  two  levels  will 
be  sufficient  for  this  purpose. 

Number  of  replications 

The  number  of  times  a  trial  for  each  display  type  is  repeated. 

Displays  tested. 

The  types  of  auditory  and  tactile  displays  tested  will  still  need 
to  be  determined.  We  have  4  auditory  and  6  tactile  designs  to 
choose  from,  but  it  will  not  be  possible  to  test  each  one  of  the 
displays  due  to  time  constraints. 

3.3.4  Summary  of  Multimodal  Display  Testing 


In  this  report,  a  number  of  different  multimodal  displays  were  proposed  to  supplement  the 
baseline  GCS.  These  displays  varied  in  their  complexity,  the  methods  of  information  presentation, 
as  well  as  many  other  dimensions.  However,  the  feasibility  and  usability  of  the  displays  must  first 
be  determined  before  they  can  be  selected  for  use  in  the  multimodal  GCS.  In  this  section,  two 
methods  for  examining  the  suitability  of  the  auditory  and  tactile  displays  proposed  were 
described:  discriminability  and  urgency.  Using  these  two  metrics,  the  proposed  designs  can  be 
tested  using  smaller  scale  experiments  before  implementation  into  the  multimodal  GCS  is 
required.  This  will  also  allow  for  testing  of  various  parameters  for  each  design.  Once  the  auditory 
and  tactile  designs  are  finalized,  they  can  be  implemented  into  the  multimodal  GCS  for  the  next 
stage  of  the  project. 


4  Conclusion 


The  work  covered  in  this  report  is  in  support  of  the  CF  JUSTAS  Project.  The  work  was 
performed  in  preparation  for  a  large  scale  study  comparing  the  two  GCS  designs.  One  of  the  GCS 
designs  will  be  a  visual  interface,  called  the  “baseline  condition”.  The  other  design  will  be  a 
multimodal  interface  where  tactile  and  auditory  information  will  be  added  to  the  interface.  The 
overall  goal  of  the  GCS  study  is  to  support  the  future  acquisition  of  a  MALE  UAV,  and 
understanding  the  human  monitoring  challenges  of  this  type  of  UAV.  Two  major  activities  were 
accomplished  and  described:  the  design  of  auditory  and  tactile  stimuli  for  the  multimodal 
condition  of  the  GCS  study,  and  preparation  and  testing  for  the  baseline  condition  of  the  GCS 
study. 

A  literature  review  was  completed  on  relevant  auditory  and  tactile  displays  in  UAV  simulation 
environments.  This  literature  was  used  to  generate  a  number  of  different  design  proposals  for 
multimodal  displays.  Two  of  these  displays,  an  engine  RPM  sonification  and  aggregate  attitude 
upset  tactile  display  were  chosen  for  further  development  and  testing.  These  two  interfaces  were 
implemented  using  the  hardware  provided  by  DRDC-Toronto.  A  proposal  for  testing  the  displays 
was  provided  as  a  starting  point  for  further  discussion. 

In  preparation  for  the  baseline  GCS  pilot  study,  ethics  documentation  from  both  the  DRDC- 
Toronto  and  University  of  Waterloo  ethics  boards  is  provided.  In  addition,  documentation  of 
extensive  testing  of  the  experimental  apparatus  was  conducted  which  included:  identifying  and 
correcting  issues  with  the  eye-tracking  setup,  identifying  bugs  with  the  GCS  simulation,  and  set¬ 
up  and  placement  of  all  apparatus  and  stimuli.  The  development  of  a  training  video  was  also 
documented.  Overall,  these  activities  have  allowed  us  to  reach  a  state  of  readiness  to  move  on  to 
the  next  stage  of  the  project. 
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Department  of  National  Defence 
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Defence  Research  &  Development  Canada 
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Director  Research  and  Development  Knowledge  and  Information 
Management 
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Research  &  Development 

